00000100:00100000:1.0:1789760689.290688:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512515200:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760689.358485:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972462 gen 3 00000100:00100000:1.0:1789760689.358520:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00514b100 x1876699512515200/t4294972463(4294972463) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760705 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760689.358559:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760689.358565:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760689.358573:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760689.358601:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760689.358624:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512515200:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760689.424681:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5145, 5145] 00000080:00200000:1.0:1789760689.424694:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760689.424700:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760689.424712:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21078016 i_blocks 43015 00000020:00200000:1.0:1789760689.424716:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760689.424724:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760689.424753:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21078016, count: 4096 00000080:00200000:1.0:1789760689.424758:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21078016, bytes: 4096 00000080:00200000:1.0:1789760689.424767:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760689.424775:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760689.424777:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21078016, 21082112) 0 0 00000080:00200000:1.0:1789760689.424781:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760689.424795:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5146, 5146] 00020000:00200000:1.0:1789760689.424803:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21078016 chunk: [21078016, 21082112) 21082112, 4096 00020000:00200000:1.0:1789760689.424960:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760689.424979:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5146, 5146] 00000008:00010000:1.0:1789760689.424995:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760689.425000:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760689.425009:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21078016, 21082112) append 0 append lockpos 0 00000080:00200000:1.0:1789760689.425021:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21078016=0x141a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760689.425065:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760689.425068:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21082112 00000020:00010000:1.0:1789760689.425076:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760689.425078:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760689.425081:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760689.425084:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760689.425086:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760689.425092:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760689.425097:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760689.425112:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:1.0:1789760689.425193:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 21078016<>21082112 00000080:00200000:1.0:1789760689.472056:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5146, 5146] 00000080:00200000:1.0:1789760689.472071:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760689.472076:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760689.472088:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21082112 i_blocks 43015 00000020:00200000:1.0:1789760689.472092:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760689.472099:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760689.472127:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21082112, count: 4096 00000080:00200000:1.0:1789760689.472133:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21082112, bytes: 4096 00000080:00200000:1.0:1789760689.472140:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760689.472148:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760689.472150:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21082112, 21086208) 0 0 00000080:00200000:1.0:1789760689.472154:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760689.472166:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5147, 5147] 00020000:00200000:1.0:1789760689.472172:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21082112 chunk: [21082112, 21086208) 21086208, 4096 00020000:00200000:1.0:1789760689.472176:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760689.472190:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5147, 5147] 00000008:00010000:1.0:1789760689.472215:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760689.472219:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760689.472229:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21082112, 21086208) append 0 append lockpos 0 00000080:00200000:1.0:1789760689.472240:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21082112=0x141b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760689.472287:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760689.472290:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21086208 00000020:00010000:1.0:1789760689.472297:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760689.472300:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760689.472303:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760689.472306:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760689.472309:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760689.472316:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760689.472321:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760689.472335:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760689.472410:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94380 - object 0x240000400:5 offset 21082112<>21086208 00000080:00200000:1.0:1789760689.514097:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5147, 5147] 00000080:00200000:1.0:1789760689.514111:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760689.514116:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760689.514128:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21086208 i_blocks 43015 00000020:00200000:1.0:1789760689.514133:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760689.514141:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760689.514169:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21086208, count: 4096 00000080:00200000:1.0:1789760689.514175:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21086208, bytes: 4096 00000080:00200000:1.0:1789760689.514184:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760689.514193:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760689.514195:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21086208, 21090304) 0 0 00000080:00200000:1.0:1789760689.514199:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760689.514223:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5148, 5148] 00020000:00200000:1.0:1789760689.514229:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21086208 chunk: [21086208, 21090304) 21090304, 4096 00020000:00200000:1.0:1789760689.514232:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760689.514248:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5148, 5148] 00000008:00010000:1.0:1789760689.514263:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760689.514268:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760689.514276:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21086208, 21090304) append 0 append lockpos 0 00000080:00200000:1.0:1789760689.514287:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21086208=0x141c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760689.514345:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760689.514348:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21090304 00000020:00010000:1.0:1789760689.514354:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760689.514357:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760689.514360:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760689.514363:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760689.514366:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760689.514371:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760689.514376:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760689.514390:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760689.514472:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 21086208<>21090304 00000080:00200000:1.0:1789760689.556970:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5148, 5148] 00000080:00200000:1.0:1789760689.556982:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760689.556987:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760689.556999:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21090304 i_blocks 43015 00000020:00200000:1.0:1789760689.557002:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760689.557011:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760689.557036:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21090304, count: 4096 00000080:00200000:1.0:1789760689.557041:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21090304, bytes: 4096 00000080:00200000:1.0:1789760689.557049:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760689.557056:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760689.557058:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21090304, 21094400) 0 0 00000080:00200000:1.0:1789760689.557061:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760689.557075:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5149, 5149] 00020000:00200000:1.0:1789760689.557081:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21090304 chunk: [21090304, 21094400) 21094400, 4096 00020000:00200000:1.0:1789760689.557085:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760689.557099:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5149, 5149] 00000008:00010000:1.0:1789760689.557113:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760689.557117:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760689.557125:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21090304, 21094400) append 0 append lockpos 0 00000080:00200000:1.0:1789760689.557137:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21090304=0x141d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760689.557180:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760689.557182:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21094400 00000020:00010000:1.0:1789760689.557188:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760689.557191:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760689.557194:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760689.557196:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760689.557199:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760689.557215:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760689.557220:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760689.557233:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:1.0:1789760689.557316:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 21090304<>21094400 00000080:00200000:1.0:1789760689.592766:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5149, 5149] 00000080:00200000:1.0:1789760689.592778:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760689.592783:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760689.592794:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21094400 i_blocks 43015 00000020:00200000:1.0:1789760689.592798:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760689.592806:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760689.592840:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21094400, count: 4096 00000080:00200000:1.0:1789760689.592949:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21094400, bytes: 4096 00000080:00200000:1.0:1789760689.592958:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760689.592964:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760689.592966:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21094400, 21098496) 0 0 00000080:00200000:1.0:1789760689.592970:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760689.592982:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5150, 5150] 00020000:00200000:1.0:1789760689.592988:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21094400 chunk: [21094400, 21098496) 21098496, 4096 00020000:00200000:1.0:1789760689.592992:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760689.593006:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5150, 5150] 00000008:00010000:1.0:1789760689.593018:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760689.593022:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760689.593032:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21094400, 21098496) append 0 append lockpos 0 00000080:00200000:1.0:1789760689.593044:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21094400=0x141e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760689.593089:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760689.593092:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21098496 00000020:00010000:1.0:1789760689.593099:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760689.593102:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760689.593105:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760689.593108:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760689.593110:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760689.593116:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760689.593121:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760689.593134:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42900 00000008:00100000:1.0:1789760689.593282:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96680 - object 0x240000400:5 offset 21094400<>21098496 00000080:00200000:1.0:1789760689.639443:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5150, 5150] 00000080:00200000:1.0:1789760689.639455:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760689.639461:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760689.639473:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21098496 i_blocks 43015 00000020:00200000:1.0:1789760689.639477:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760689.639496:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760689.639522:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21098496, count: 4096 00000080:00200000:1.0:1789760689.639529:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21098496, bytes: 4096 00000080:00200000:1.0:1789760689.639536:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760689.639543:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760689.639545:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21098496, 21102592) 0 0 00000080:00200000:1.0:1789760689.639549:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760689.639562:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5151, 5151] 00020000:00200000:1.0:1789760689.639568:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21098496 chunk: [21098496, 21102592) 21102592, 4096 00020000:00200000:1.0:1789760689.639573:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760689.639586:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5151, 5151] 00000008:00010000:1.0:1789760689.639600:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760689.639604:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760689.639613:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21098496, 21102592) append 0 append lockpos 0 00000080:00200000:1.0:1789760689.639623:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21098496=0x141f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760689.639668:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760689.639672:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21102592 00000020:00010000:1.0:1789760689.639678:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760689.639681:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760689.639684:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760689.639687:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760689.639690:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760689.639695:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760689.639700:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760689.639714:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f426c0 00000008:00100000:1.0:1789760689.639785:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97480 - object 0x240000400:5 offset 21098496<>21102592 00000080:00200000:1.0:1789760689.676128:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5151, 5151] 00000080:00200000:1.0:1789760689.676141:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760689.676145:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760689.676157:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21102592 i_blocks 43015 00000020:00200000:1.0:1789760689.676160:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760689.676168:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760689.676197:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21102592, count: 4096 00000080:00200000:1.0:1789760689.676264:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21102592, bytes: 4096 00000080:00200000:1.0:1789760689.676273:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760689.676280:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760689.676282:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21102592, 21106688) 0 0 00000080:00200000:1.0:1789760689.676285:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760689.676298:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5152, 5152] 00020000:00200000:1.0:1789760689.676305:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21102592 chunk: [21102592, 21106688) 21106688, 4096 00020000:00200000:1.0:1789760689.676309:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760689.676323:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5152, 5152] 00000008:00010000:1.0:1789760689.676335:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760689.676339:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760689.676347:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21102592, 21106688) append 0 append lockpos 0 00000080:00200000:1.0:1789760689.676357:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21102592=0x1420000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760689.676399:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760689.676402:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21106688 00000020:00010000:1.0:1789760689.676409:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760689.676412:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760689.676415:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760689.676418:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760689.676420:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760689.676425:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760689.676430:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760689.676444:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f429c0 00000008:00100000:1.0:1789760689.676579:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95880 - object 0x240000400:5 offset 21102592<>21106688 00000080:00200000:1.0:1789760689.713119:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5152, 5152] 00000080:00200000:1.0:1789760689.713131:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760689.713136:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760689.713147:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21106688 i_blocks 43015 00000020:00200000:1.0:1789760689.713151:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760689.713159:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760689.713185:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21106688, count: 4096 00000080:00200000:1.0:1789760689.713190:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21106688, bytes: 4096 00000080:00200000:1.0:1789760689.713373:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760689.713381:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760689.713383:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21106688, 21110784) 0 0 00000080:00200000:1.0:1789760689.713387:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760689.713400:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5153, 5153] 00020000:00200000:1.0:1789760689.713407:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21106688 chunk: [21106688, 21110784) 21110784, 4096 00020000:00200000:1.0:1789760689.713410:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760689.713425:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5153, 5153] 00000008:00010000:1.0:1789760689.713439:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760689.713443:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760689.713452:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21106688, 21110784) append 0 append lockpos 0 00000080:00200000:1.0:1789760689.713462:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21106688=0x1421000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760689.713571:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760689.713575:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21110784 00000020:00010000:1.0:1789760689.713583:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760689.713586:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760689.713589:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760689.713592:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760689.713595:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760689.713602:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760689.713607:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760689.713621:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760689.713700:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96d80 - object 0x240000400:5 offset 21106688<>21110784 00000080:00200000:1.0:1789760689.752835:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5153, 5153] 00000080:00200000:1.0:1789760689.752848:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760689.752853:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760689.752864:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21110784 i_blocks 43015 00000020:00200000:1.0:1789760689.752868:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760689.752876:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760689.752936:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21110784, count: 4096 00000080:00200000:1.0:1789760689.752943:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21110784, bytes: 4096 00000080:00200000:1.0:1789760689.752952:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760689.752960:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760689.752962:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21110784, 21114880) 0 0 00000080:00200000:1.0:1789760689.752966:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760689.752980:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5154, 5154] 00020000:00200000:1.0:1789760689.752988:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21110784 chunk: [21110784, 21114880) 21114880, 4096 00020000:00200000:1.0:1789760689.752992:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760689.753007:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5154, 5154] 00000008:00010000:1.0:1789760689.753020:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760689.753025:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760689.753034:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21110784, 21114880) append 0 append lockpos 0 00000080:00200000:1.0:1789760689.753045:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21110784=0x1422000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760689.753091:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760689.753095:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21114880 00000020:00010000:1.0:1789760689.753103:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760689.753107:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760689.753110:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760689.753113:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760689.753116:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760689.753122:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760689.753127:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760689.753140:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760689.753333:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95880 - object 0x240000400:5 offset 21110784<>21114880 00000080:00200000:1.0:1789760689.791264:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5154, 5154] 00000080:00200000:1.0:1789760689.791280:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760689.791285:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760689.791296:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21114880 i_blocks 43015 00000020:00200000:1.0:1789760689.791299:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760689.791307:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760689.791332:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21114880, count: 4096 00000080:00200000:1.0:1789760689.791338:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21114880, bytes: 4096 00000080:00200000:1.0:1789760689.791346:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760689.791353:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760689.791355:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21114880, 21118976) 0 0 00000080:00200000:1.0:1789760689.791358:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760689.791371:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5155, 5155] 00020000:00200000:1.0:1789760689.791378:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21114880 chunk: [21114880, 21118976) 21118976, 4096 00020000:00200000:1.0:1789760689.791381:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760689.791397:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5155, 5155] 00000008:00010000:1.0:1789760689.791411:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760689.791416:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760689.791425:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21114880, 21118976) append 0 append lockpos 0 00000080:00200000:1.0:1789760689.791437:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21114880=0x1423000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760689.791551:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760689.791554:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21118976 00000020:00010000:1.0:1789760689.791562:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760689.791565:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760689.791568:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760689.791571:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760689.791574:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760689.791580:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760689.791586:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760689.791601:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f429c0 00000008:00100000:1.0:1789760689.791680:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97480 - object 0x240000400:5 offset 21114880<>21118976 00010000:00010000:1.0:1789760689.949067:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1789760690.162743:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5160, 5160] 00000080:00200000:1.0:1789760690.162755:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760690.162760:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760690.162771:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21139456 i_blocks 43015 00000020:00200000:1.0:1789760690.162775:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760690.162784:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760690.162892:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21139456, count: 4096 00000080:00200000:1.0:1789760690.162925:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21139456, bytes: 4096 00000080:00200000:1.0:1789760690.162935:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760690.162942:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760690.162944:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21139456, 21143552) 0 0 00000080:00200000:1.0:1789760690.162948:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760690.162961:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5161, 5161] 00020000:00200000:1.0:1789760690.162967:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21139456 chunk: [21139456, 21143552) 21143552, 4096 00020000:00200000:1.0:1789760690.162971:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760690.162986:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5161, 5161] 00000008:00010000:1.0:1789760690.163000:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760690.163004:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760690.163013:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21139456, 21143552) append 0 append lockpos 0 00000080:00200000:1.0:1789760690.163023:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21139456=0x1429000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760690.163071:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760690.163074:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21143552 00000020:00010000:1.0:1789760690.163080:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760690.163083:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.163086:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.163089:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760690.163091:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760690.163097:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760690.163102:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760690.163115:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f426c0 00000008:00100000:1.0:1789760690.163196:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96680 - object 0x240000400:5 offset 21139456<>21143552 00000100:00100000:1.0:1789760690.163294:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760690.163311:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512517376:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760690.189362:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972480 gen 3 00000008:00010000:1.0:1789760690.189394:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f426c0 00000020:00200000:1.0:1789760690.189400:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760690.189406:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760690.189462:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f426c0 00000100:00100000:1.0:1789760690.189482:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512517376:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760690.220555:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5162, 5162] 00000080:00200000:1.0:1789760690.220569:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760690.220575:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760690.220587:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21147648 i_blocks 43015 00000020:00200000:1.0:1789760690.220590:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760690.220598:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760690.220622:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21147648, count: 4096 00000080:00200000:1.0:1789760690.220628:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21147648, bytes: 4096 00000080:00200000:1.0:1789760690.220637:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760690.220646:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760690.220648:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21147648, 21151744) 0 0 00000080:00200000:1.0:1789760690.220652:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760690.220666:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5163, 5163] 00020000:00200000:1.0:1789760690.220673:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21147648 chunk: [21147648, 21151744) 21151744, 4096 00020000:00200000:1.0:1789760690.220677:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760690.220690:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5163, 5163] 00000008:00010000:1.0:1789760690.220705:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760690.220710:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760690.220720:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21147648, 21151744) append 0 append lockpos 0 00000080:00200000:1.0:1789760690.220732:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21147648=0x142b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760690.220777:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760690.220780:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21151744 00000020:00010000:1.0:1789760690.220787:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760690.220790:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.220793:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.220796:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760690.220799:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760690.220817:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760690.220823:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760690.220836:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f426c0 00000008:00100000:1.0:1789760690.221017:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96680 - object 0x240000400:5 offset 21147648<>21151744 00000080:00200000:1.0:1789760690.248947:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5163, 5163] 00000080:00200000:1.0:1789760690.248959:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760690.248964:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760690.248974:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21151744 i_blocks 43015 00000020:00200000:1.0:1789760690.248977:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760690.248985:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760690.249019:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21151744, count: 4096 00000080:00200000:1.0:1789760690.249110:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21151744, bytes: 4096 00000080:00200000:1.0:1789760690.249119:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760690.249127:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760690.249130:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21151744, 21155840) 0 0 00000080:00200000:1.0:1789760690.249133:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760690.249145:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5164, 5164] 00020000:00200000:1.0:1789760690.249153:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21151744 chunk: [21151744, 21155840) 21155840, 4096 00020000:00200000:1.0:1789760690.249156:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760690.249173:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5164, 5164] 00000008:00010000:1.0:1789760690.249190:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760690.249195:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760690.249266:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21151744, 21155840) append 0 append lockpos 0 00000080:00200000:1.0:1789760690.249279:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21151744=0x142c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760690.249324:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760690.249327:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21155840 00000020:00010000:1.0:1789760690.249334:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760690.249337:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.249339:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.249343:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760690.249346:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760690.249352:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760690.249359:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760690.249375:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42900 00000008:00100000:1.0:1789760690.249454:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 21151744<>21155840 00000080:00200000:1.0:1789760690.283152:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5164, 5164] 00000080:00200000:1.0:1789760690.283165:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760690.283169:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760690.283179:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21155840 i_blocks 43015 00000020:00200000:1.0:1789760690.283183:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760690.283191:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760690.283283:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21155840, count: 4096 00000080:00200000:1.0:1789760690.283291:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21155840, bytes: 4096 00000080:00200000:1.0:1789760690.283299:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760690.283307:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760690.283309:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21155840, 21159936) 0 0 00000080:00200000:1.0:1789760690.283313:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760690.283326:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5165, 5165] 00020000:00200000:1.0:1789760690.283334:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21155840 chunk: [21155840, 21159936) 21159936, 4096 00020000:00200000:1.0:1789760690.283338:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760690.283353:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5165, 5165] 00000008:00010000:1.0:1789760690.283367:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760690.283372:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760690.283380:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21155840, 21159936) append 0 append lockpos 0 00000080:00200000:1.0:1789760690.283392:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21155840=0x142d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760690.283441:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760690.283444:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21159936 00000020:00010000:1.0:1789760690.283451:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760690.283454:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.283457:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.283460:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760690.283462:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760690.283468:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760690.283474:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760690.283696:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:1.0:1789760690.283785:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 21155840<>21159936 00000080:00200000:1.0:1789760690.311746:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5165, 5165] 00000080:00200000:1.0:1789760690.311759:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760690.311765:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760690.311779:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21159936 i_blocks 43015 00000020:00200000:1.0:1789760690.311782:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760690.311800:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760690.311833:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21159936, count: 4096 00000080:00200000:1.0:1789760690.311839:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21159936, bytes: 4096 00000080:00200000:1.0:1789760690.311846:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760690.311854:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760690.311856:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21159936, 21164032) 0 0 00000080:00200000:1.0:1789760690.311860:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760690.311874:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5166, 5166] 00020000:00200000:1.0:1789760690.311880:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21159936 chunk: [21159936, 21164032) 21164032, 4096 00020000:00200000:1.0:1789760690.311884:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760690.311926:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5166, 5166] 00000008:00010000:1.0:1789760690.311940:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760690.311945:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760690.311955:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21159936, 21164032) append 0 append lockpos 0 00000080:00200000:1.0:1789760690.311964:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21159936=0x142e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760690.312009:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760690.312012:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21164032 00000020:00010000:1.0:1789760690.312018:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760690.312021:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.312024:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.312027:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760690.312029:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760690.312036:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760690.312041:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760690.312056:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760690.312128:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94380 - object 0x240000400:5 offset 21159936<>21164032 00000080:00200000:1.0:1789760690.360743:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5166, 5166] 00000080:00200000:1.0:1789760690.360756:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760690.360760:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760690.360775:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21164032 i_blocks 43015 00000020:00200000:1.0:1789760690.360778:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760690.360785:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760690.360831:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21164032, count: 4096 00000080:00200000:1.0:1789760690.360837:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21164032, bytes: 4096 00000080:00200000:1.0:1789760690.360845:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760690.360853:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760690.360855:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21164032, 21168128) 0 0 00000080:00200000:1.0:1789760690.360858:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760690.360871:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5167, 5167] 00020000:00200000:1.0:1789760690.360877:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21164032 chunk: [21164032, 21168128) 21168128, 4096 00020000:00200000:1.0:1789760690.360881:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760690.360895:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5167, 5167] 00000008:00010000:1.0:1789760690.360932:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760690.360937:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760690.360947:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21164032, 21168128) append 0 append lockpos 0 00000080:00200000:1.0:1789760690.360957:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21164032=0x142f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760690.361002:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760690.361005:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21168128 00000020:00010000:1.0:1789760690.361011:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760690.361015:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.361017:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.361020:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760690.361023:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760690.361029:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760690.361036:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760690.361049:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760690.361123:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 21164032<>21168128 00000080:00200000:1.0:1789760690.399236:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5167, 5167] 00000080:00200000:1.0:1789760690.399248:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760690.399253:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760690.399266:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21168128 i_blocks 43015 00000020:00200000:1.0:1789760690.399270:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760690.399278:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760690.399305:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21168128, count: 4096 00000080:00200000:1.0:1789760690.399311:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21168128, bytes: 4096 00000080:00200000:1.0:1789760690.399318:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760690.399325:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760690.399327:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21168128, 21172224) 0 0 00000080:00200000:1.0:1789760690.399331:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760690.399342:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5168, 5168] 00020000:00200000:1.0:1789760690.399348:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21168128 chunk: [21168128, 21172224) 21172224, 4096 00020000:00200000:1.0:1789760690.399352:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760690.399365:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5168, 5168] 00000008:00010000:1.0:1789760690.399378:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760690.399383:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760690.399391:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21168128, 21172224) append 0 append lockpos 0 00000080:00200000:1.0:1789760690.399400:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21168128=0x1430000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760690.399439:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760690.399441:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21172224 00000020:00010000:1.0:1789760690.399447:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760690.399450:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.399453:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.399455:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760690.399458:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760690.399712:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760690.399718:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760690.399734:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:1.0:1789760690.399823:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 21168128<>21172224 00000080:00200000:1.0:1789760690.462204:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5168, 5168] 00000080:00200000:1.0:1789760690.462218:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760690.462222:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760690.462235:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21172224 i_blocks 43015 00000020:00200000:1.0:1789760690.462239:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760690.462247:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760690.462275:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21172224, count: 4096 00000080:00200000:1.0:1789760690.462282:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21172224, bytes: 4096 00000080:00200000:1.0:1789760690.462291:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760690.462299:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760690.462301:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21172224, 21176320) 0 0 00000080:00200000:1.0:1789760690.462305:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760690.462316:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5169, 5169] 00020000:00200000:1.0:1789760690.462322:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21172224 chunk: [21172224, 21176320) 21176320, 4096 00020000:00200000:1.0:1789760690.462326:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760690.462340:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5169, 5169] 00000008:00010000:1.0:1789760690.462355:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760690.462360:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760690.462370:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21172224, 21176320) append 0 append lockpos 0 00000080:00200000:1.0:1789760690.462381:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21172224=0x1431000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760690.462425:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760690.462427:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21176320 00000020:00010000:1.0:1789760690.462434:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760690.462437:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.462440:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.462444:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760690.462447:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760690.462453:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760690.462459:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760690.462473:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42900 00000008:00100000:1.0:1789760690.462616:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94700 - object 0x240000400:5 offset 21172224<>21176320 00000080:00200000:1.0:1789760690.559983:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5169, 5169] 00000080:00200000:1.0:1789760690.559998:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760690.560003:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760690.560015:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21176320 i_blocks 43015 00000020:00200000:1.0:1789760690.560018:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760690.560027:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760690.560056:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21176320, count: 4096 00000080:00200000:1.0:1789760690.560063:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21176320, bytes: 4096 00000080:00200000:1.0:1789760690.560070:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760690.560078:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760690.560080:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21176320, 21180416) 0 0 00000080:00200000:1.0:1789760690.560084:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760690.560097:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5170, 5170] 00020000:00200000:1.0:1789760690.560105:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21176320 chunk: [21176320, 21180416) 21180416, 4096 00020000:00200000:1.0:1789760690.560109:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760690.560123:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5170, 5170] 00000008:00010000:1.0:1789760690.560136:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760690.560141:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760690.560150:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21176320, 21180416) append 0 append lockpos 0 00000080:00200000:1.0:1789760690.560161:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21176320=0x1432000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760690.560204:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760690.560206:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21180416 00000020:00010000:1.0:1789760690.560213:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760690.560216:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.560219:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.560223:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760690.560225:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760690.560233:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760690.560238:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760690.560253:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f426c0 00000008:00100000:1.0:1789760690.560327:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 21176320<>21180416 00000080:00200000:1.0:1789760690.619765:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5170, 5170] 00000080:00200000:1.0:1789760690.619778:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760690.619783:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760690.619794:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21180416 i_blocks 43015 00000020:00200000:1.0:1789760690.619798:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760690.619817:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760690.619841:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21180416, count: 4096 00000080:00200000:1.0:1789760690.619847:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21180416, bytes: 4096 00000080:00200000:1.0:1789760690.619855:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760690.619863:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760690.619865:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21180416, 21184512) 0 0 00000080:00200000:1.0:1789760690.619869:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760690.619882:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5171, 5171] 00020000:00200000:1.0:1789760690.619888:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21180416 chunk: [21180416, 21184512) 21184512, 4096 00020000:00200000:1.0:1789760690.619892:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760690.619933:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5171, 5171] 00000008:00010000:1.0:1789760690.619948:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760690.619952:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760690.619961:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21180416, 21184512) append 0 append lockpos 0 00000080:00200000:1.0:1789760690.619972:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21180416=0x1433000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760690.620013:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760690.620016:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21184512 00000020:00010000:1.0:1789760690.620022:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760690.620026:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.620029:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.620032:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760690.620035:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760690.620041:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760690.620047:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760690.620060:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f429c0 00000008:00100000:1.0:1789760690.620147:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 21180416<>21184512 00000080:00200000:1.0:1789760690.666358:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5171, 5171] 00000080:00200000:1.0:1789760690.666370:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760690.666375:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760690.666387:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21184512 i_blocks 43015 00000020:00200000:1.0:1789760690.666391:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760690.666399:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760690.666423:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21184512, count: 4096 00000080:00200000:1.0:1789760690.666429:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21184512, bytes: 4096 00000080:00200000:1.0:1789760690.666436:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760690.666443:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760690.666445:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21184512, 21188608) 0 0 00000080:00200000:1.0:1789760690.666448:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760690.666461:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5172, 5172] 00020000:00200000:1.0:1789760690.666468:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21184512 chunk: [21184512, 21188608) 21188608, 4096 00020000:00200000:1.0:1789760690.666471:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760690.666545:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5172, 5172] 00000008:00010000:1.0:1789760690.666559:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760690.666564:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760690.666573:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21184512, 21188608) append 0 append lockpos 0 00000080:00200000:1.0:1789760690.666584:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21184512=0x1434000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760690.666628:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760690.666631:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21188608 00000020:00010000:1.0:1789760690.666638:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760690.666640:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.666643:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760690.666646:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760690.666648:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760690.666654:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760690.666660:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760690.666673:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760690.666749:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95c00 - object 0x240000400:5 offset 21184512<>21188608 00000100:00100000:1.0:1789760692.022553:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512523136:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760692.060618:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972525 gen 3 00000008:00010000:1.0:1789760692.060722:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760692.060730:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760692.060742:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760692.060831:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760692.060856:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512523136:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760692.064278:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512523264:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760692.171854:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972526 gen 3 00000008:00010000:1.0:1789760692.171889:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760692.171896:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760692.171936:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760692.171957:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760692.171978:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512523264:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760692.176098:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760692.176131:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512523392:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760692.261088:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972527 gen 3 00000008:00010000:1.0:1789760692.261123:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760692.261129:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760692.261138:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760692.261147:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760692.261169:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512523392:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760692.263323:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512523520:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760692.351190:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760692.351202:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760692.351212:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760692.351265:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760692.351290:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512523520:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760692.420934:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5211, 5211] 00000080:00200000:1.0:1789760692.420952:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760692.420957:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760692.420968:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21348352 i_blocks 43015 00000020:00200000:1.0:1789760692.420973:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760692.420984:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760692.421011:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21348352, count: 4096 00000080:00200000:1.0:1789760692.421017:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21348352, bytes: 4096 00000080:00200000:1.0:1789760692.421027:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760692.421034:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760692.421036:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21348352, 21352448) 0 0 00000080:00200000:1.0:1789760692.421040:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760692.421053:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5212, 5212] 00020000:00200000:1.0:1789760692.421059:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21348352 chunk: [21348352, 21352448) 21352448, 4096 00020000:00200000:1.0:1789760692.421062:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760692.421092:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5212, 5212] 00000008:00010000:1.0:1789760692.421108:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760692.421113:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760692.421121:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21348352, 21352448) append 0 append lockpos 0 00000080:00200000:1.0:1789760692.421132:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21348352=0x145c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760692.421179:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760692.421182:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21352448 00000020:00010000:1.0:1789760692.421189:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760692.421192:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760692.421206:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c128/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760692.421209:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760692.421212:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760692.421220:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760692.421226:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760692.421245:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760692.421323:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 21348352<>21352448 00000100:00100000:1.0:1789760692.421373:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760692.421390:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512523904:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760692.459257:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972531 gen 3 00000008:00010000:1.0:1789760692.459297:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760692.459303:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760692.459311:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760692.459334:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760692.459359:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512523904:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760693.161846:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5224, 5224] 00000080:00200000:1.0:1789760693.161857:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760693.161860:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760693.161871:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21401600 i_blocks 43015 00000020:00200000:1.0:1789760693.161873:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760693.161879:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760693.161933:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21401600, count: 4096 00000080:00200000:1.0:1789760693.161939:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21401600, bytes: 4096 00000080:00200000:1.0:1789760693.161948:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760693.161955:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760693.161957:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21401600, 21405696) 0 0 00000080:00200000:1.0:1789760693.161960:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760693.161971:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5225, 5225] 00020000:00200000:1.0:1789760693.161977:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21401600 chunk: [21401600, 21405696) 21405696, 4096 00020000:00200000:1.0:1789760693.161980:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760693.161992:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5225, 5225] 00000008:00010000:1.0:1789760693.162004:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760693.162008:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760693.162014:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21401600, 21405696) append 0 append lockpos 0 00000080:00200000:1.0:1789760693.162022:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21401600=0x1469000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760693.162059:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760693.162061:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21405696 00000020:00010000:1.0:1789760693.162065:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760693.162068:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760693.162070:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c128/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760693.162072:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760693.162074:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760693.162078:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760693.162082:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760693.162093:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760693.162153:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 21401600<>21405696 00000100:00100000:1.0:1789760693.377524:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760693.377539:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512526464:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760693.434815:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972548 gen 3 00000008:00010000:1.0:1789760693.434852:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760693.434858:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760693.434867:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760693.434888:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760693.434948:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512526464:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760693.435843:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512526592:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760693.488448:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972549 gen 3 00000008:00010000:1.0:1789760693.488485:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760693.488492:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760693.488501:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760693.488521:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760693.488546:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512526592:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760693.489404:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760693.489430:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512526720:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760693.549948:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972550 gen 3 00000008:00010000:1.0:1789760693.549984:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760693.549990:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760693.549999:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760693.550026:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760693.550049:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512526720:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760693.550722:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512526848:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760693.635613:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760693.635629:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760693.635640:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760693.635672:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760693.635695:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512526848:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760693.959220:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512527360:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760694.037864:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972555 gen 3 00000008:00010000:1.0:1789760694.037977:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760694.037987:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760694.037997:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760694.038034:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760694.038065:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512527360:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760694.184446:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760694.184468:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512527744:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760694.261897:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972558 gen 3 00000008:00010000:1.0:1789760694.261971:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760694.261977:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760694.261986:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760694.262004:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760694.262025:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512527744:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760694.262767:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512527872:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760694.390601:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760694.390612:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760694.390621:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760694.390696:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760694.390722:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512527872:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760694.397147:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760694.397186:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512528000:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760694.509833:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972559 gen 3 00000100:00100000:1.0:1789760694.509866:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00514a300 x1876699512528000/t4294972560(4294972560) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760710 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760694.509929:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760694.509935:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760694.509944:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760694.509963:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760694.509982:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512528000:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760694.550443:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972561 gen 3 00000008:00010000:1.0:1789760694.550501:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760694.550508:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760694.550518:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760694.550535:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760694.550557:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512528128:192.168.203.156@tcp:4:dd.0 00000020:00200000:3.0:1789760694.551718:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760694.551727:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760694.551753:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21475328, count: 4096 00000080:00200000:3.0:1789760694.551760:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21475328, bytes: 4096 00000080:00200000:3.0:1789760694.551768:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760694.551974:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760694.551977:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21475328, 21479424) 0 0 00000080:00200000:3.0:1789760694.551981:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760694.551994:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5243, 5243] 00020000:00200000:3.0:1789760694.552002:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21475328 chunk: [21475328, 21479424) 21479424, 4096 00020000:00200000:3.0:1789760694.552005:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760694.552021:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5243, 5243] 00000008:00010000:3.0:1789760694.552035:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760694.552041:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760694.552050:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21475328, 21479424) append 0 append lockpos 0 00000080:00200000:3.0:1789760694.552061:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21475328=0x147b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760694.552107:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760694.552110:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21479424 00000020:00010000:3.0:1789760694.552117:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760694.552120:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.552123:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c128/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.552126:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760694.552128:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760694.552135:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760694.552140:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760694.552156:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760694.552249:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af800 - object 0x240000400:5 offset 21475328<>21479424 00000100:00100000:3.0:1789760694.552292:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760694.552309:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512528256:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760694.598059:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5243, 5243] 00000080:00200000:3.0:1789760694.598072:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760694.598079:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760694.598091:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21479424 i_blocks 43015 00000020:00200000:3.0:1789760694.598094:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760694.598102:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760694.598129:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21479424, count: 4096 00000080:00200000:3.0:1789760694.598134:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21479424, bytes: 4096 00000080:00200000:3.0:1789760694.598142:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760694.598149:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760694.598151:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21479424, 21483520) 0 0 00000080:00200000:3.0:1789760694.598154:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760694.598230:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5244, 5244] 00020000:00200000:3.0:1789760694.598237:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21479424 chunk: [21479424, 21483520) 21483520, 4096 00020000:00200000:3.0:1789760694.598240:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760694.598254:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5244, 5244] 00000008:00010000:3.0:1789760694.598268:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760694.598273:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760694.598282:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21479424, 21483520) append 0 append lockpos 0 00000080:00200000:3.0:1789760694.598294:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21479424=0x147c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760694.598336:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760694.598339:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21483520 00000020:00010000:3.0:1789760694.598346:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760694.598349:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.598352:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c128/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760694.598355:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760694.598357:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760694.598363:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760694.598369:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760694.598383:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760694.598530:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 21479424<>21483520 00000080:00200000:3.0:1789760694.666717:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5244, 5244] 00000080:00200000:3.0:1789760694.666731:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760694.666736:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:3.0:1789760694.667465:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512528512:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760694.759711:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512528640:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760694.829004:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760694.829016:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760694.829354:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760694.829375:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760694.829399:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512528640:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760694.831363:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512528768:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760694.876781:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972566 gen 3 00000008:00010000:1.0:1789760694.876830:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760694.876836:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760694.876844:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760694.876861:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760694.876881:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512528768:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760694.877370:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512528896:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760694.927191:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972567 gen 3 00000008:00010000:1.0:1789760694.927226:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760694.927233:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760694.927243:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760694.927262:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760694.927283:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512528896:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760695.066819:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5250, 5250] 00000080:00200000:3.0:1789760695.066832:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760695.066838:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760695.066849:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21508096 i_blocks 43015 00000020:00200000:3.0:1789760695.066852:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760695.066860:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760695.066885:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21508096, count: 4096 00000080:00200000:3.0:1789760695.066892:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21508096, bytes: 4096 00000080:00200000:3.0:1789760695.066922:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760695.066930:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760695.066932:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21508096, 21512192) 0 0 00000080:00200000:3.0:1789760695.066936:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760695.066949:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5251, 5251] 00020000:00200000:3.0:1789760695.066957:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21508096 chunk: [21508096, 21512192) 21512192, 4096 00020000:00200000:3.0:1789760695.066960:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760695.066975:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5251, 5251] 00000008:00010000:3.0:1789760695.066988:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760695.066993:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760695.067002:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21508096, 21512192) append 0 append lockpos 0 00000080:00200000:3.0:1789760695.067013:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21508096=0x1483000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760695.067056:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760695.067059:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21512192 00000020:00010000:3.0:1789760695.067065:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760695.067068:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.067071:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.067074:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760695.067077:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760695.067085:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760695.067090:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760695.067104:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760695.067294:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae680 - object 0x240000400:5 offset 21508096<>21512192 00000100:00100000:3.0:1789760695.067330:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760695.067343:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512529280:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760695.121423:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760695.121442:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512529408:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760695.203214:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972571 gen 3 00000008:00010000:1.0:1789760695.203264:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760695.203271:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760695.203280:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760695.203300:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760695.203322:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512529408:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760695.204284:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760695.204300:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512529536:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760695.257683:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972572 gen 3 00000008:00010000:1.0:1789760695.257720:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760695.257727:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760695.257737:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760695.257759:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760695.257799:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512529536:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760695.258380:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000008:00100000:1.0:1789760695.258458:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 21520384<>21524480 00000100:00100000:1.0:1789760695.258503:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512529664:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760695.341811:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972573 gen 3 00000008:00010000:1.0:1789760695.341845:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760695.341852:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760695.341861:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760695.341882:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760695.341944:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512529664:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760695.342632:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512529792:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760695.414835:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972574 gen 3 00000008:00010000:1.0:1789760695.414884:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760695.414890:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760695.414898:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760695.414940:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760695.414959:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512529792:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760695.611559:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5257, 5257] 00000080:00200000:3.0:1789760695.611572:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760695.611577:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760695.611589:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21536768 i_blocks 43015 00000020:00200000:3.0:1789760695.611592:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760695.611600:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760695.611625:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21536768, count: 4096 00000080:00200000:3.0:1789760695.611631:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21536768, bytes: 4096 00000080:00200000:3.0:1789760695.611639:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760695.611646:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760695.611648:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21536768, 21540864) 0 0 00000080:00200000:3.0:1789760695.611652:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760695.611665:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5258, 5258] 00020000:00200000:3.0:1789760695.611672:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21536768 chunk: [21536768, 21540864) 21540864, 4096 00020000:00200000:3.0:1789760695.611675:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760695.611689:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5258, 5258] 00000008:00010000:3.0:1789760695.611702:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760695.611707:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760695.611716:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21536768, 21540864) append 0 append lockpos 0 00000080:00200000:3.0:1789760695.611725:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21536768=0x148a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760695.611777:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760695.611780:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21540864 00000020:00010000:3.0:1789760695.611786:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760695.611789:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.611792:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.611795:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760695.611797:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760695.611804:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760695.611810:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760695.611831:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760695.611947:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 21536768<>21540864 00000100:00100000:3.0:1789760695.611980:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760695.611991:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512530176:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760695.658896:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5258, 5258] 00000080:00200000:3.0:1789760695.658931:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760695.658936:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760695.658948:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21540864 i_blocks 43015 00000020:00200000:3.0:1789760695.658951:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760695.658959:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760695.658983:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21540864, count: 4096 00000080:00200000:3.0:1789760695.658989:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21540864, bytes: 4096 00000080:00200000:3.0:1789760695.658997:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760695.659004:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760695.659006:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21540864, 21544960) 0 0 00000080:00200000:3.0:1789760695.659010:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760695.659024:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5259, 5259] 00020000:00200000:3.0:1789760695.659031:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21540864 chunk: [21540864, 21544960) 21544960, 4096 00020000:00200000:3.0:1789760695.659035:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760695.659050:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5259, 5259] 00000008:00010000:3.0:1789760695.659064:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760695.659068:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760695.659076:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21540864, 21544960) append 0 append lockpos 0 00000080:00200000:3.0:1789760695.659086:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21540864=0x148b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760695.659126:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760695.659128:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21544960 00000020:00010000:3.0:1789760695.659134:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760695.659137:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.659140:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.659143:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760695.659145:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760695.659151:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760695.659168:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760695.659181:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760695.659252:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad880 - object 0x240000400:5 offset 21540864<>21544960 00000080:00200000:3.0:1789760695.697071:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5259, 5259] 00000080:00200000:3.0:1789760695.697085:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760695.697090:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760695.697102:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21544960 i_blocks 43015 00000020:00200000:3.0:1789760695.697106:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760695.697114:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760695.697139:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21544960, count: 4096 00000080:00200000:3.0:1789760695.697146:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21544960, bytes: 4096 00000080:00200000:3.0:1789760695.697154:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760695.697169:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760695.697172:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21544960, 21549056) 0 0 00000080:00200000:3.0:1789760695.697176:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760695.697190:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5260, 5260] 00020000:00200000:3.0:1789760695.697196:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21544960 chunk: [21544960, 21549056) 21549056, 4096 00020000:00200000:3.0:1789760695.697200:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760695.697215:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5260, 5260] 00000008:00010000:3.0:1789760695.697229:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760695.697233:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760695.697241:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21544960, 21549056) append 0 append lockpos 0 00000080:00200000:3.0:1789760695.697252:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21544960=0x148c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760695.697293:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760695.697296:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21549056 00000020:00010000:3.0:1789760695.697302:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760695.697306:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.697309:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.697312:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760695.697314:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760695.697321:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760695.697327:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760695.697341:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760695.697431:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adf80 - object 0x240000400:5 offset 21544960<>21549056 00000080:00200000:3.0:1789760695.749110:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5260, 5260] 00000080:00200000:3.0:1789760695.749124:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760695.749129:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760695.749141:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21549056 i_blocks 43015 00000020:00200000:3.0:1789760695.749144:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760695.749151:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760695.749188:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21549056, count: 4096 00000080:00200000:3.0:1789760695.749194:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21549056, bytes: 4096 00000080:00200000:3.0:1789760695.749201:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760695.749207:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760695.749209:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21549056, 21553152) 0 0 00000080:00200000:3.0:1789760695.749213:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760695.749227:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5261, 5261] 00020000:00200000:3.0:1789760695.749233:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21549056 chunk: [21549056, 21553152) 21553152, 4096 00020000:00200000:3.0:1789760695.749237:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760695.749250:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5261, 5261] 00000008:00010000:3.0:1789760695.749264:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760695.749268:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760695.749277:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21549056, 21553152) append 0 append lockpos 0 00000080:00200000:3.0:1789760695.749287:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21549056=0x148d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760695.749327:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760695.749330:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21553152 00000020:00010000:3.0:1789760695.749336:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760695.749340:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.749343:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.749346:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760695.749349:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760695.749356:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760695.749362:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760695.749375:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760695.749456:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad180 - object 0x240000400:5 offset 21549056<>21553152 00000080:00200000:3.0:1789760695.791773:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5261, 5261] 00000080:00200000:3.0:1789760695.791786:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760695.791792:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760695.791803:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21553152 i_blocks 43015 00000020:00200000:3.0:1789760695.791807:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760695.791814:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760695.791838:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21553152, count: 4096 00000080:00200000:3.0:1789760695.791845:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21553152, bytes: 4096 00000080:00200000:3.0:1789760695.791852:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760695.791859:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760695.791861:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21553152, 21557248) 0 0 00000080:00200000:3.0:1789760695.791875:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760695.791890:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5262, 5262] 00020000:00200000:3.0:1789760695.791898:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21553152 chunk: [21553152, 21557248) 21557248, 4096 00020000:00200000:3.0:1789760695.791922:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760695.791938:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5262, 5262] 00000008:00010000:3.0:1789760695.791952:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760695.791956:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760695.791966:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21553152, 21557248) append 0 append lockpos 0 00000080:00200000:3.0:1789760695.791976:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21553152=0x148e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760695.792019:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760695.792022:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21557248 00000020:00010000:3.0:1789760695.792027:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760695.792031:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.792034:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.792038:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760695.792040:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760695.792047:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760695.792051:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760695.792062:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760695.792133:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af100 - object 0x240000400:5 offset 21553152<>21557248 00000080:00200000:3.0:1789760695.833514:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5262, 5262] 00000080:00200000:3.0:1789760695.833526:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760695.833531:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760695.833541:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21557248 i_blocks 43015 00000020:00200000:3.0:1789760695.833545:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760695.833552:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760695.833576:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21557248, count: 4096 00000080:00200000:3.0:1789760695.833582:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21557248, bytes: 4096 00000080:00200000:3.0:1789760695.833590:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760695.833596:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760695.833598:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21557248, 21561344) 0 0 00000080:00200000:3.0:1789760695.833602:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760695.833613:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5263, 5263] 00020000:00200000:3.0:1789760695.833620:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21557248 chunk: [21557248, 21561344) 21561344, 4096 00020000:00200000:3.0:1789760695.833624:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760695.833640:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5263, 5263] 00000008:00010000:3.0:1789760695.833653:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760695.833658:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760695.833666:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21557248, 21561344) append 0 append lockpos 0 00000080:00200000:3.0:1789760695.833677:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21557248=0x148f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760695.833719:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760695.833722:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21561344 00000020:00010000:3.0:1789760695.833729:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760695.833732:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.833735:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.833739:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760695.833741:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760695.833747:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760695.833753:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760695.833777:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760695.833858:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aea00 - object 0x240000400:5 offset 21557248<>21561344 00000080:00200000:3.0:1789760695.879988:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5263, 5263] 00000080:00200000:3.0:1789760695.880000:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760695.880005:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760695.880017:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21561344 i_blocks 43015 00000020:00200000:3.0:1789760695.880020:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760695.880027:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760695.880053:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21561344, count: 4096 00000080:00200000:3.0:1789760695.880058:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21561344, bytes: 4096 00000080:00200000:3.0:1789760695.880067:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760695.880073:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760695.880075:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21561344, 21565440) 0 0 00000080:00200000:3.0:1789760695.880079:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760695.880091:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5264, 5264] 00020000:00200000:3.0:1789760695.880097:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21561344 chunk: [21561344, 21565440) 21565440, 4096 00020000:00200000:3.0:1789760695.880100:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760695.880114:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5264, 5264] 00000008:00010000:3.0:1789760695.880127:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760695.880132:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760695.880140:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21561344, 21565440) append 0 append lockpos 0 00000080:00200000:3.0:1789760695.880151:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21561344=0x1490000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760695.880209:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760695.880212:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21565440 00000020:00010000:3.0:1789760695.880218:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760695.880221:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.880224:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760695.880227:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760695.880230:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760695.880237:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760695.880242:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760695.880257:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760695.880344:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac380 - object 0x240000400:5 offset 21561344<>21565440 00000100:00100000:3.0:1789760695.923023:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972583 gen 3 00000008:00010000:3.0:1789760695.923058:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760695.923066:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760695.923074:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760695.923094:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760695.923118:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512530944:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760695.923710:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760695.923723:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512531072:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760695.967219:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972584 gen 3 00000008:00010000:3.0:1789760695.967255:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760695.967262:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760695.967270:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760695.967287:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760695.967310:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512531072:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760695.968090:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512531200:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760696.013828:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972585 gen 3 00000008:00010000:3.0:1789760696.013863:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760696.013869:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.013878:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.013896:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760696.013943:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512531200:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760696.014551:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760696.014565:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512531328:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760696.051617:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972586 gen 3 00000008:00010000:3.0:1789760696.051653:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760696.051660:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.051668:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.051755:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760696.051800:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512531328:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760696.052741:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512531456:192.168.203.156@tcp:4:dd.0 00010000:00010000:3.0:1789760696.093119:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760696.093425:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972587 gen 3 00000008:00010000:3.0:1789760696.093465:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760696.093471:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.093479:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.093496:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760696.093516:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512531456:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760696.094143:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760696.094399:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512531584:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760696.126931:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760696.126942:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.126951:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.126969:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760696.126992:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512531584:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760696.127621:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512531712:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760696.161389:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972589 gen 3 00000008:00010000:3.0:1789760696.161436:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760696.161443:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.161451:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.161506:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760696.161531:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512531712:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760696.162749:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760696.162775:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512531840:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760696.214412:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972590 gen 3 00000008:00010000:3.0:1789760696.214458:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760696.214465:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.214473:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.214492:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760696.214517:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512531840:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760696.215299:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512531968:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760696.262609:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972591 gen 3 00000008:00010000:3.0:1789760696.262643:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760696.262649:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.262656:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.262675:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760696.262697:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512531968:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760696.263646:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760696.263660:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512532096:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760696.327827:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972592 gen 3 00000008:00010000:3.0:1789760696.328081:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760696.328089:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.328098:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.328115:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760696.328139:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512532096:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760696.387631:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760696.387641:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.387650:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.387668:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760696.387690:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512532224:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760696.389419:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512532352:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760696.435214:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972594 gen 3 00000008:00010000:3.0:1789760696.435260:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760696.435267:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.435274:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.435297:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760696.435322:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512532352:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760696.436248:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512532480:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760696.487314:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972595 gen 3 00000008:00010000:1.0:1789760696.487350:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760696.487357:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760696.487366:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760696.487389:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760696.487411:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512532480:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760696.488081:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760696.488094:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512532608:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760696.534824:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972596 gen 3 00000008:00010000:1.0:1789760696.534868:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760696.534875:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760696.534885:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760696.534934:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760696.534954:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512532608:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760696.536069:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512532736:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760696.575584:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972597 gen 3 00000008:00010000:1.0:1789760696.575615:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760696.575622:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760696.575631:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760696.575653:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760696.575675:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512532736:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760696.576522:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760696.576534:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512532864:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760696.610828:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972598 gen 3 00000008:00010000:1.0:1789760696.610865:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760696.610871:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760696.610880:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760696.610931:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760696.610954:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512532864:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760696.612249:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512532992:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760696.663630:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972599 gen 3 00000008:00010000:1.0:1789760696.663663:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760696.663669:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760696.663677:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760696.663696:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760696.663716:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512532992:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760696.664575:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512533120:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760696.722172:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972600 gen 3 00000008:00010000:1.0:1789760696.722208:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760696.722214:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760696.722223:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760696.722243:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760696.722265:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512533120:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760696.723278:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512533248:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760696.772186:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972601 gen 3 00000008:00010000:1.0:1789760696.772221:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760696.772229:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760696.772250:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760696.772270:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760696.772293:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512533248:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760696.772829:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512533376:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760696.825193:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972602 gen 3 00000008:00010000:1.0:1789760696.825230:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760696.825306:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760696.825318:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760696.825391:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760696.825415:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512533376:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760696.826754:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5283, 5283] 00000080:00200000:3.0:1789760696.826781:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760696.826786:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760696.828057:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21643264 i_blocks 43015 00000020:00200000:3.0:1789760696.828063:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760696.828074:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760696.828102:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21643264, count: 4096 00000080:00200000:3.0:1789760696.828110:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21643264, bytes: 4096 00000080:00200000:3.0:1789760696.828121:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760696.828129:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760696.828131:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21643264, 21647360) 0 0 00000080:00200000:3.0:1789760696.828134:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760696.828158:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5284, 5284] 00020000:00200000:3.0:1789760696.828166:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21643264 chunk: [21643264, 21647360) 21647360, 4096 00020000:00200000:3.0:1789760696.828169:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760696.828186:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5284, 5284] 00000008:00010000:3.0:1789760696.828199:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760696.828204:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760696.828213:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21643264, 21647360) append 0 append lockpos 0 00000080:00200000:3.0:1789760696.828225:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21643264=0x14a4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760696.828276:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760696.828279:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21647360 00000020:00010000:3.0:1789760696.828285:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760696.828288:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760696.828291:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760696.828294:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760696.828296:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760696.828303:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760696.828309:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760696.828329:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760696.828423:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac380 - object 0x240000400:5 offset 21643264<>21647360 00000100:00100000:3.0:1789760696.828585:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760696.828601:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512533504:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760696.897224:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760696.897248:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760696.897257:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760696.897277:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760696.897299:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512533504:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760696.897580:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5284, 5284] 00000080:00200000:1.0:1789760696.897592:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760696.897598:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760696.897609:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21647360 i_blocks 43015 00000020:00200000:1.0:1789760696.897613:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760696.897621:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760696.897646:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21647360, count: 4096 00000080:00200000:1.0:1789760696.897651:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21647360, bytes: 4096 00000080:00200000:1.0:1789760696.897659:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760696.897666:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760696.897667:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21647360, 21651456) 0 0 00000080:00200000:1.0:1789760696.897672:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760696.897685:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5285, 5285] 00020000:00200000:1.0:1789760696.897692:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21647360 chunk: [21647360, 21651456) 21651456, 4096 00020000:00200000:1.0:1789760696.897695:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760696.897710:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5285, 5285] 00000008:00010000:1.0:1789760696.897723:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760696.897728:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760696.897737:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21647360, 21651456) append 0 append lockpos 0 00000080:00200000:1.0:1789760696.897747:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21647360=0x14a5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760696.897801:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760696.897804:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21651456 00000020:00010000:1.0:1789760696.897810:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760696.897823:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760696.897827:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760696.897829:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760696.897832:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760696.897838:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760696.897844:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760696.897856:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f429c0 00000008:00100000:1.0:1789760696.897974:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 21647360<>21651456 00000100:00100000:1.0:1789760696.898001:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760696.898012:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512533632:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760696.950112:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972604 gen 3 00000008:00010000:1.0:1789760696.950156:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f429c0 00000020:00200000:1.0:1789760696.950163:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760696.950171:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760696.950194:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f429c0 00000100:00100000:1.0:1789760696.950216:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512533632:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760696.950369:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5285, 5285] 00000080:00200000:3.0:1789760696.950381:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760696.950386:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760696.950398:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21651456 i_blocks 43015 00000020:00200000:3.0:1789760696.950401:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760696.950409:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760696.950433:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21651456, count: 4096 00000080:00200000:3.0:1789760696.950440:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21651456, bytes: 4096 00000080:00200000:3.0:1789760696.950448:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760696.950455:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760696.950457:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21651456, 21655552) 0 0 00000080:00200000:3.0:1789760696.950460:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760696.950473:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5286, 5286] 00020000:00200000:3.0:1789760696.950479:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21651456 chunk: [21651456, 21655552) 21655552, 4096 00020000:00200000:3.0:1789760696.950483:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760696.950497:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5286, 5286] 00000008:00010000:3.0:1789760696.950509:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760696.950514:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760696.950522:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21651456, 21655552) append 0 append lockpos 0 00000080:00200000:3.0:1789760696.950532:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21651456=0x14a6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760696.950571:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760696.950574:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21655552 00000020:00010000:3.0:1789760696.950581:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760696.950584:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760696.950587:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760696.950590:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760696.950593:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760696.950599:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760696.950605:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760696.950618:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760696.950692:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af480 - object 0x240000400:5 offset 21651456<>21655552 00000100:00100000:3.0:1789760697.016827:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972605 gen 3 00000008:00010000:3.0:1789760697.016865:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760697.016871:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.016880:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.016914:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760697.016934:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512533760:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760697.017189:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5286, 5286] 00000080:00200000:1.0:1789760697.017200:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760697.017205:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760697.017217:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21655552 i_blocks 43015 00000020:00200000:1.0:1789760697.017220:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760697.017227:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760697.017259:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21655552, count: 4096 00000080:00200000:1.0:1789760697.017265:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21655552, bytes: 4096 00000080:00200000:1.0:1789760697.017273:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760697.017279:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760697.017281:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21655552, 21659648) 0 0 00000080:00200000:1.0:1789760697.017285:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760697.017296:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5287, 5287] 00020000:00200000:1.0:1789760697.017302:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21655552 chunk: [21655552, 21659648) 21659648, 4096 00020000:00200000:1.0:1789760697.017306:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760697.017319:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5287, 5287] 00000008:00010000:1.0:1789760697.017331:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760697.017335:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760697.017343:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21655552, 21659648) append 0 append lockpos 0 00000080:00200000:1.0:1789760697.017355:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21655552=0x14a7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760697.017395:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760697.017407:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21659648 00000020:00010000:1.0:1789760697.017413:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760697.017416:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760697.017419:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760697.017422:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760697.017425:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760697.017431:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760697.017435:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760697.017447:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f429c0 00000008:00100000:1.0:1789760697.017515:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 21655552<>21659648 00000100:00100000:1.0:1789760697.055406:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972606 gen 3 00000008:00010000:1.0:1789760697.055441:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f429c0 00000020:00200000:1.0:1789760697.055447:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760697.055455:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760697.055481:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f429c0 00000100:00100000:1.0:1789760697.055504:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512533888:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760697.055604:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5287, 5287] 00000080:00200000:3.0:1789760697.055617:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760697.055622:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760697.055634:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21659648 i_blocks 43015 00000020:00200000:3.0:1789760697.055638:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760697.055645:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760697.055669:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21659648, count: 4096 00000080:00200000:3.0:1789760697.055676:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21659648, bytes: 4096 00000080:00200000:3.0:1789760697.055683:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760697.055690:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760697.055692:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21659648, 21663744) 0 0 00000080:00200000:3.0:1789760697.055696:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760697.055707:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5288, 5288] 00020000:00200000:3.0:1789760697.055714:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21659648 chunk: [21659648, 21663744) 21663744, 4096 00020000:00200000:3.0:1789760697.055717:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760697.055732:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5288, 5288] 00000008:00010000:3.0:1789760697.055745:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760697.055749:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760697.055771:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21659648, 21663744) append 0 append lockpos 0 00000080:00200000:3.0:1789760697.055783:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21659648=0x14a8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760697.055826:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760697.055829:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21663744 00000020:00010000:3.0:1789760697.055836:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760697.055839:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760697.055842:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760697.055846:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760697.055848:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760697.056010:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760697.056018:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760697.056033:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760697.056106:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af480 - object 0x240000400:5 offset 21659648<>21663744 00000100:00100000:3.0:1789760697.116215:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972607 gen 3 00000008:00010000:3.0:1789760697.116256:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760697.116264:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.116471:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.117891:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760697.117931:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512534016:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760697.121168:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5288, 5288] 00000080:00200000:1.0:1789760697.121189:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760697.121195:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760697.121209:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21663744 i_blocks 43015 00000020:00200000:1.0:1789760697.121214:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760697.121227:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760697.121256:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21663744, count: 4096 00000080:00200000:1.0:1789760697.121263:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21663744, bytes: 4096 00000080:00200000:1.0:1789760697.121498:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760697.121507:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760697.121509:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21663744, 21667840) 0 0 00000080:00200000:1.0:1789760697.121512:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760697.121526:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5289, 5289] 00020000:00200000:1.0:1789760697.121534:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21663744 chunk: [21663744, 21667840) 21667840, 4096 00020000:00200000:1.0:1789760697.121537:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760697.121554:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5289, 5289] 00000008:00010000:1.0:1789760697.121568:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760697.121573:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760697.121583:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21663744, 21667840) append 0 append lockpos 0 00000080:00200000:1.0:1789760697.121595:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21663744=0x14a9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760697.121826:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760697.121829:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21667840 00000020:00010000:1.0:1789760697.121836:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760697.121839:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760697.121842:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760697.121845:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760697.121848:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760697.122660:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760697.122669:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760697.122688:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f429c0 00000008:00100000:1.0:1789760697.122785:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 21663744<>21667840 00000100:00100000:1.0:1789760697.122840:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760697.122871:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512534144:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760697.262299:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f429c0 00000020:00200000:1.0:1789760697.262311:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760697.262319:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760697.262341:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f429c0 00000100:00100000:1.0:1789760697.262363:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512534144:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760697.358363:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5290, 5290] 00000080:00200000:3.0:1789760697.358376:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760697.358381:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760697.358393:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21671936 i_blocks 43015 00000020:00200000:3.0:1789760697.358397:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760697.358405:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760697.358431:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21671936, count: 4096 00000080:00200000:3.0:1789760697.358436:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21671936, bytes: 4096 00000080:00200000:3.0:1789760697.358445:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760697.358451:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760697.358454:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21671936, 21676032) 0 0 00000080:00200000:3.0:1789760697.358457:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760697.358468:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5291, 5291] 00020000:00200000:3.0:1789760697.358474:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21671936 chunk: [21671936, 21676032) 21676032, 4096 00020000:00200000:3.0:1789760697.358478:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760697.358492:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5291, 5291] 00000008:00010000:3.0:1789760697.358505:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760697.358510:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760697.358519:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21671936, 21676032) append 0 append lockpos 0 00000080:00200000:3.0:1789760697.358529:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21671936=0x14ab000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760697.358573:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760697.358576:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21676032 00000020:00010000:3.0:1789760697.358583:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760697.358586:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760697.358589:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc018e42758/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760697.358592:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760697.358595:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760697.358602:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760697.358607:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760697.358623:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760697.358697:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af480 - object 0x240000400:5 offset 21671936<>21676032 00000100:00100000:3.0:1789760697.459363:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972610 gen 3 00000008:00010000:3.0:1789760697.459414:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760697.459420:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.459430:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.459452:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760697.459485:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512534400:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760697.538406:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512534656:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760697.593775:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972612 gen 3 00000008:00010000:3.0:1789760697.593812:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760697.593818:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.593827:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.593851:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760697.593876:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512534656:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760697.597581:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512534784:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760697.724624:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972613 gen 3 00000008:00010000:3.0:1789760697.724660:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760697.724667:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.724676:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.724693:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760697.724716:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512534784:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760697.725323:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760697.725336:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512534912:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760697.807175:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972614 gen 3 00000008:00010000:3.0:1789760697.807211:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760697.807219:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.807227:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.807246:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760697.807271:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512534912:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760697.807942:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512535040:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760697.864559:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760697.864573:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.864584:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.864603:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760697.864632:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512535040:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760697.865392:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512535168:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760697.913582:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972616 gen 3 00000008:00010000:3.0:1789760697.913646:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760697.913655:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760697.913665:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760697.918325:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760697.918385:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512535168:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760697.919812:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512535296:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760698.027274:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972617 gen 3 00000008:00010000:3.0:1789760698.027331:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760698.027340:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760698.027352:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760698.027381:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760698.027424:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512535296:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760698.029080:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760698.029111:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512535424:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760698.146533:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512535680:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760698.146690:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2360:1876699512535808:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760698.164542:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972618 gen 3 00000008:00010000:3.0:1789760698.164597:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760698.164606:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760698.164618:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760698.164641:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760698.164673:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512535424:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760698.173773:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e6cf7267-9b19-462a-a263-14115df15dab:2360:1876699512535552:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760698.188487:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2360:1876699512535808:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760698.188544:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512535680:192.168.203.156@tcp:400:kworker.0 00000100:00100000:1.0:1789760698.242558:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760698.242654:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512536064:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760698.311657:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760698.311671:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760698.311681:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760698.311694:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760698.311718:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512536064:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760698.405416:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972621 gen 3 00000008:00010000:1.0:1789760698.405484:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760698.405493:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760698.405505:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760698.405525:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760698.405554:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512536192:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760698.593148:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512536704:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760698.770163:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21745664 i_blocks 43015 00000020:00200000:1.0:1789760698.770168:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760698.770179:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760698.770213:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21745664, count: 4096 00000080:00200000:1.0:1789760698.770220:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21745664, bytes: 4096 00000080:00200000:1.0:1789760698.770231:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760698.770241:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760698.770243:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21745664, 21749760) 0 0 00000080:00200000:1.0:1789760698.770247:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760698.770265:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5309, 5309] 00020000:00200000:1.0:1789760698.770274:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21745664 chunk: [21745664, 21749760) 21749760, 4096 00020000:00200000:1.0:1789760698.770280:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760698.770299:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5309, 5309] 00000008:00010000:1.0:1789760698.770316:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760698.770322:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760698.770336:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21745664, 21749760) append 0 append lockpos 0 00000080:00200000:1.0:1789760698.770350:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21745664=0x14bd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760698.770409:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760698.770412:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21749760 00000020:00010000:1.0:1789760698.770421:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760698.770424:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760698.770428:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760698.770431:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760698.770434:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760698.770442:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760698.770449:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760698.770498:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f429c0 00000008:00100000:1.0:1789760698.770615:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97100 - object 0x240000400:5 offset 21745664<>21749760 00000080:00200000:1.0:1789760698.823019:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5309, 5309] 00000080:00200000:1.0:1789760698.823035:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760698.823041:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760698.823056:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21749760 i_blocks 43015 00000020:00200000:1.0:1789760698.823059:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760698.823068:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760698.823102:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21749760, count: 4096 00000080:00200000:1.0:1789760698.823110:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21749760, bytes: 4096 00000080:00200000:1.0:1789760698.823119:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760698.823128:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760698.823130:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21749760, 21753856) 0 0 00000080:00200000:1.0:1789760698.823135:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760698.823152:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5310, 5310] 00020000:00200000:1.0:1789760698.823160:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21749760 chunk: [21749760, 21753856) 21753856, 4096 00020000:00200000:1.0:1789760698.823165:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760698.823185:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5310, 5310] 00000008:00010000:1.0:1789760698.823202:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760698.823207:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760698.823218:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21749760, 21753856) append 0 append lockpos 0 00000080:00200000:1.0:1789760698.823232:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21749760=0x14be000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760698.823279:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760698.823282:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21753856 00000020:00010000:1.0:1789760698.823290:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760698.823294:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760698.823297:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760698.823300:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760698.823302:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760698.823309:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760698.823316:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760698.870258:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5310, 5310] 00000080:00200000:1.0:1789760698.870276:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760698.870281:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760698.870296:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21753856 i_blocks 43015 00000020:00200000:1.0:1789760698.870300:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760698.870309:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760698.870340:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21753856, count: 4096 00000080:00200000:1.0:1789760698.870347:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21753856, bytes: 4096 00000080:00200000:1.0:1789760698.870356:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760698.870365:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760698.870367:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21753856, 21757952) 0 0 00000080:00200000:1.0:1789760698.870372:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760698.870390:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5311, 5311] 00020000:00200000:1.0:1789760698.870398:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21753856 chunk: [21753856, 21757952) 21757952, 4096 00020000:00200000:1.0:1789760698.870402:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760698.870433:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5311, 5311] 00000008:00010000:1.0:1789760698.870453:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760698.870565:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760698.870579:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21753856, 21757952) append 0 append lockpos 0 00000080:00200000:1.0:1789760698.870594:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21753856=0x14bf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760698.870658:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760698.870662:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21757952 00000020:00010000:1.0:1789760698.870670:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760698.870674:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760698.870677:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760698.870681:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760698.870684:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760698.870693:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760698.870700:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760698.870716:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:1.0:1789760698.870855:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 21753856<>21757952 00000100:00100000:1.0:1789760698.870928:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760698.870946:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512537344:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760698.916385:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972630 gen 3 00000008:00010000:1.0:1789760698.916435:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42240 00000020:00200000:1.0:1789760698.916443:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760698.916454:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760698.916480:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42240 00000100:00100000:1.0:1789760698.916512:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512537344:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760698.947936:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5312, 5312] 00000080:00200000:1.0:1789760698.947956:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760698.947962:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760698.947977:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21762048 i_blocks 43015 00000020:00200000:1.0:1789760698.947982:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760698.947993:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760698.948028:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21762048, count: 4096 00000080:00200000:1.0:1789760698.948039:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21762048, bytes: 4096 00000080:00200000:1.0:1789760698.948050:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760698.948060:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760698.948062:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21762048, 21766144) 0 0 00000080:00200000:1.0:1789760698.948067:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760698.948084:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5313, 5313] 00020000:00200000:1.0:1789760698.948093:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21762048 chunk: [21762048, 21766144) 21766144, 4096 00020000:00200000:1.0:1789760698.948098:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760698.948117:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5313, 5313] 00000008:00010000:1.0:1789760698.948145:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760698.948151:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760698.948162:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21762048, 21766144) append 0 append lockpos 0 00000080:00200000:1.0:1789760698.948178:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21762048=0x14c1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760698.948298:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760698.948302:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21766144 00000020:00010000:1.0:1789760698.948311:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760698.948315:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760698.948318:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760698.948322:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760698.948325:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760698.948333:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760698.948341:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760698.948359:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:1.0:1789760698.948620:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 21762048<>21766144 00000080:00200000:1.0:1789760699.052583:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5313, 5313] 00000080:00200000:1.0:1789760699.052601:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760699.052608:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760699.052624:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21766144 i_blocks 43015 00000020:00200000:1.0:1789760699.052629:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760699.052639:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760699.052680:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21766144, count: 4096 00000080:00200000:1.0:1789760699.052690:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21766144, bytes: 4096 00000080:00200000:1.0:1789760699.052705:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760699.052716:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760699.052718:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21766144, 21770240) 0 0 00000080:00200000:1.0:1789760699.052723:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760699.053693:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5314, 5314] 00020000:00200000:1.0:1789760699.053710:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21766144 chunk: [21766144, 21770240) 21770240, 4096 00020000:00200000:1.0:1789760699.053715:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760699.053882:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5314, 5314] 00000008:00010000:1.0:1789760699.053920:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760699.053927:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760699.053940:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21766144, 21770240) append 0 append lockpos 0 00000080:00200000:1.0:1789760699.053957:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21766144=0x14c2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760699.054014:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760699.054017:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21770240 00000020:00010000:1.0:1789760699.054025:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760699.054028:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760699.054031:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760699.054035:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760699.054038:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760699.054048:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760699.054056:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760699.094945:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5314, 5314] 00000080:00200000:1.0:1789760699.094965:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760699.094972:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760699.094989:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21770240 i_blocks 43015 00000020:00200000:1.0:1789760699.094994:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760699.095004:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760699.095040:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21770240, count: 4096 00000080:00200000:1.0:1789760699.095116:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21770240, bytes: 4096 00000080:00200000:1.0:1789760699.095188:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760699.095198:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760699.095200:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21770240, 21774336) 0 0 00000080:00200000:1.0:1789760699.095205:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760699.095224:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5315, 5315] 00020000:00200000:1.0:1789760699.095232:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21770240 chunk: [21770240, 21774336) 21774336, 4096 00020000:00200000:1.0:1789760699.095237:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760699.095259:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5315, 5315] 00000008:00010000:1.0:1789760699.095278:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760699.095284:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760699.095297:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21770240, 21774336) append 0 append lockpos 0 00000080:00200000:1.0:1789760699.095311:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21770240=0x14c3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760699.095379:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760699.095381:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21774336 00000020:00010000:1.0:1789760699.095390:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760699.095393:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760699.095397:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760699.095400:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760699.095403:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760699.095479:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760699.095488:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760699.095517:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760699.095624:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 21770240<>21774336 00010000:00010000:3.0:1789760699.165387:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:1.0:1789760699.190508:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5315, 5315] 00000080:00200000:1.0:1789760699.190528:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760699.190535:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760699.190553:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21774336 i_blocks 43015 00000020:00200000:1.0:1789760699.190557:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760699.190569:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760699.190604:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21774336, count: 4096 00000080:00200000:1.0:1789760699.190612:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21774336, bytes: 4096 00000080:00200000:1.0:1789760699.190626:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760699.190636:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760699.190638:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21774336, 21778432) 0 0 00000080:00200000:1.0:1789760699.190644:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760699.190662:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5316, 5316] 00020000:00200000:1.0:1789760699.190672:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21774336 chunk: [21774336, 21778432) 21778432, 4096 00020000:00200000:1.0:1789760699.190677:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760699.190698:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5316, 5316] 00000008:00010000:1.0:1789760699.190718:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760699.190724:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760699.190738:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21774336, 21778432) append 0 append lockpos 0 00000080:00200000:1.0:1789760699.190762:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21774336=0x14c4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760699.190833:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760699.190836:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21778432 00000020:00010000:1.0:1789760699.190844:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760699.190847:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760699.190850:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760699.190853:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760699.190857:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760699.190864:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760699.190870:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760699.190890:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:1.0:1789760699.191034:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97100 - object 0x240000400:5 offset 21774336<>21778432 00000080:00200000:1.0:1789760699.295124:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5316, 5316] 00000080:00200000:1.0:1789760699.295144:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760699.295150:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760699.295166:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21778432 i_blocks 43015 00000020:00200000:1.0:1789760699.295169:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760699.295183:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760699.295220:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21778432, count: 4096 00000080:00200000:1.0:1789760699.295229:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21778432, bytes: 4096 00000080:00200000:1.0:1789760699.295243:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760699.295252:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760699.295254:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21778432, 21782528) 0 0 00000080:00200000:1.0:1789760699.295260:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760699.295277:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5317, 5317] 00020000:00200000:1.0:1789760699.295285:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21778432 chunk: [21778432, 21782528) 21782528, 4096 00020000:00200000:1.0:1789760699.295289:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760699.295445:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5317, 5317] 00000008:00010000:1.0:1789760699.295476:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760699.295483:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760699.295496:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21778432, 21782528) append 0 append lockpos 0 00000080:00200000:1.0:1789760699.295562:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21778432=0x14c5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760699.295629:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760699.295632:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21782528 00000020:00010000:1.0:1789760699.295641:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760699.295645:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760699.295649:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760699.295663:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760699.295668:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760699.295683:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760699.295691:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760699.295712:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42900 00000008:00100000:1.0:1789760699.295833:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 21778432<>21782528 00000080:00200000:3.0:1789760699.355228:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5317, 5317] 00000080:00200000:3.0:1789760699.355241:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.355247:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.355258:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21782528 i_blocks 43015 00000020:00200000:3.0:1789760699.355262:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.355270:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.355295:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21782528, count: 4096 00000080:00200000:3.0:1789760699.355302:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21782528, bytes: 4096 00000080:00200000:3.0:1789760699.355310:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.355318:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.355320:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21782528, 21786624) 0 0 00000080:00200000:3.0:1789760699.355323:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.355336:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5318, 5318] 00020000:00200000:3.0:1789760699.355341:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21782528 chunk: [21782528, 21786624) 21786624, 4096 00020000:00200000:3.0:1789760699.355345:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.355360:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5318, 5318] 00000008:00010000:3.0:1789760699.355373:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.355378:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760699.355387:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21782528, 21786624) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.355398:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21782528=0x14c6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.355532:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.355535:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21786624 00000020:00010000:3.0:1789760699.355542:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.355546:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.355548:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.355551:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.355554:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.355561:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.355566:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.355582:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760699.355658:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac000 - object 0x240000400:5 offset 21782528<>21786624 00000100:00100000:3.0:1789760699.355709:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760699.355725:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512538240:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760699.408383:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5318, 5318] 00000080:00200000:3.0:1789760699.408397:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.408402:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.408416:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21786624 i_blocks 43015 00000020:00200000:3.0:1789760699.408420:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.408428:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.408455:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21786624, count: 4096 00000080:00200000:3.0:1789760699.408462:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21786624, bytes: 4096 00000080:00200000:3.0:1789760699.408470:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.408477:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.408479:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21786624, 21790720) 0 0 00000080:00200000:3.0:1789760699.408483:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.408498:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5319, 5319] 00020000:00200000:3.0:1789760699.408505:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21786624 chunk: [21786624, 21790720) 21790720, 4096 00020000:00200000:3.0:1789760699.408509:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.408524:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5319, 5319] 00000008:00010000:3.0:1789760699.408538:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.408543:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760699.408552:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21786624, 21790720) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.408565:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21786624=0x14c7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.408613:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.408615:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21790720 00000020:00010000:3.0:1789760699.408622:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.408625:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.408628:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.408631:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.408633:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.408639:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.408645:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.408663:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760699.408752:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae300 - object 0x240000400:5 offset 21786624<>21790720 00000080:00200000:3.0:1789760699.462281:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5319, 5319] 00000080:00200000:3.0:1789760699.462294:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.462298:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.462309:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21790720 i_blocks 43015 00000020:00200000:3.0:1789760699.462313:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.462320:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.462345:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21790720, count: 4096 00000080:00200000:3.0:1789760699.462351:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21790720, bytes: 4096 00000080:00200000:3.0:1789760699.462360:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.462367:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.462369:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21790720, 21794816) 0 0 00000080:00200000:3.0:1789760699.462374:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.462387:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5320, 5320] 00020000:00200000:3.0:1789760699.462393:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21790720 chunk: [21790720, 21794816) 21794816, 4096 00020000:00200000:3.0:1789760699.462397:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.462411:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5320, 5320] 00000008:00010000:3.0:1789760699.462426:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.462430:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760699.462439:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21790720, 21794816) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.462450:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21790720=0x14c8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.462493:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.462496:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21794816 00000020:00010000:3.0:1789760699.462502:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.462505:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.462508:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.462511:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.462514:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.462522:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.462527:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.462542:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760699.462623:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af480 - object 0x240000400:5 offset 21790720<>21794816 00000080:00200000:3.0:1789760699.510477:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5320, 5320] 00000080:00200000:3.0:1789760699.510488:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.510492:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.510501:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21794816 i_blocks 43015 00000020:00200000:3.0:1789760699.510504:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.510510:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.510534:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21794816, count: 4096 00000080:00200000:3.0:1789760699.510539:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21794816, bytes: 4096 00000080:00200000:3.0:1789760699.510546:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.510552:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.510553:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21794816, 21798912) 0 0 00000080:00200000:3.0:1789760699.510557:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.510568:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5321, 5321] 00020000:00200000:3.0:1789760699.510574:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21794816 chunk: [21794816, 21798912) 21798912, 4096 00020000:00200000:3.0:1789760699.510577:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.510588:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5321, 5321] 00000008:00010000:3.0:1789760699.510599:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.510602:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760699.510610:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21794816, 21798912) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.510619:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21794816=0x14c9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.510653:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.510655:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21798912 00000020:00010000:3.0:1789760699.510661:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.510664:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.510666:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.510669:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.510671:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.510677:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.510684:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.510698:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760699.510800:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac700 - object 0x240000400:5 offset 21794816<>21798912 00000080:00200000:3.0:1789760699.554048:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5321, 5321] 00000080:00200000:3.0:1789760699.554060:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.554065:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.554077:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21798912 i_blocks 43015 00000020:00200000:3.0:1789760699.554080:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.554088:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.554115:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21798912, count: 4096 00000080:00200000:3.0:1789760699.554121:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21798912, bytes: 4096 00000080:00200000:3.0:1789760699.554196:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.554204:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.554206:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21798912, 21803008) 0 0 00000080:00200000:3.0:1789760699.554210:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.554224:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5322, 5322] 00020000:00200000:3.0:1789760699.554230:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21798912 chunk: [21798912, 21803008) 21803008, 4096 00020000:00200000:3.0:1789760699.554234:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.554252:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5322, 5322] 00000008:00010000:3.0:1789760699.554265:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.554270:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760699.554280:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21798912, 21803008) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.554293:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21798912=0x14ca000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.554338:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.554341:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21803008 00000020:00010000:3.0:1789760699.554347:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.554351:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.554353:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.554356:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.554359:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.554367:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.554372:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.554398:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760699.554485:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 21798912<>21803008 00000080:00200000:3.0:1789760699.608551:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5322, 5322] 00000080:00200000:3.0:1789760699.608565:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.608570:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.608581:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21803008 i_blocks 43015 00000020:00200000:3.0:1789760699.608585:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.608593:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.608620:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21803008, count: 4096 00000080:00200000:3.0:1789760699.608626:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21803008, bytes: 4096 00000080:00200000:3.0:1789760699.608634:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.608641:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.608643:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21803008, 21807104) 0 0 00000080:00200000:3.0:1789760699.608647:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.608659:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5323, 5323] 00020000:00200000:3.0:1789760699.608665:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21803008 chunk: [21803008, 21807104) 21807104, 4096 00020000:00200000:3.0:1789760699.608669:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.608684:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5323, 5323] 00000008:00010000:3.0:1789760699.608697:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.608701:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760699.608709:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21803008, 21807104) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.608719:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21803008=0x14cb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.608774:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.608777:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21807104 00000020:00010000:3.0:1789760699.608784:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.608796:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.608801:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.608803:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.608806:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.608813:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.608818:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.608936:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760699.609014:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107afb80 - object 0x240000400:5 offset 21803008<>21807104 00000080:00200000:3.0:1789760699.660876:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5323, 5323] 00000080:00200000:3.0:1789760699.660889:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.660895:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.660937:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21807104 i_blocks 43015 00000020:00200000:3.0:1789760699.660941:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.660949:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.660976:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21807104, count: 4096 00000080:00200000:3.0:1789760699.660982:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21807104, bytes: 4096 00000080:00200000:3.0:1789760699.660989:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.660996:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.660998:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21807104, 21811200) 0 0 00000080:00200000:3.0:1789760699.661002:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.661015:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5324, 5324] 00020000:00200000:3.0:1789760699.661021:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21807104 chunk: [21807104, 21811200) 21811200, 4096 00020000:00200000:3.0:1789760699.661025:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.661041:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5324, 5324] 00000008:00010000:3.0:1789760699.661055:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.661060:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760699.661069:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21807104, 21811200) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.661079:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21807104=0x14cc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.661123:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.661136:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21811200 00000020:00010000:3.0:1789760699.661144:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.661147:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.661150:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.661152:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.661155:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.661161:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.661167:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.661181:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760699.661260:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 21807104<>21811200 00000080:00200000:3.0:1789760699.705426:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5324, 5324] 00000080:00200000:3.0:1789760699.705453:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.705460:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.705493:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21811200 i_blocks 43015 00000020:00200000:3.0:1789760699.705500:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.705512:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.705548:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21811200, count: 4096 00000080:00200000:3.0:1789760699.705557:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21811200, bytes: 4096 00000080:00200000:3.0:1789760699.705569:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.705578:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.705580:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21811200, 21815296) 0 0 00000080:00200000:3.0:1789760699.705584:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.705602:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5325, 5325] 00020000:00200000:3.0:1789760699.705610:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21811200 chunk: [21811200, 21815296) 21815296, 4096 00020000:00200000:3.0:1789760699.705615:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.705638:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5325, 5325] 00000008:00010000:3.0:1789760699.705659:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.705665:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760699.705678:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21811200, 21815296) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.705693:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21811200=0x14cd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.705768:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.705771:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21815296 00000020:00010000:3.0:1789760699.705780:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.705783:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.705786:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.705789:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.705792:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.705800:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.705806:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.705828:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760699.705989:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad500 - object 0x240000400:5 offset 21811200<>21815296 00000080:00200000:3.0:1789760699.783946:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5325, 5325] 00000080:00200000:3.0:1789760699.783966:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.783976:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.783992:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21815296 i_blocks 43015 00000020:00200000:3.0:1789760699.783998:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.784007:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.784043:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21815296, count: 4096 00000080:00200000:3.0:1789760699.784050:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21815296, bytes: 4096 00000080:00200000:3.0:1789760699.784058:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.784067:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.784069:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21815296, 21819392) 0 0 00000080:00200000:3.0:1789760699.784073:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.784091:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5326, 5326] 00020000:00200000:3.0:1789760699.784100:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21815296 chunk: [21815296, 21819392) 21819392, 4096 00020000:00200000:3.0:1789760699.784106:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.784126:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5326, 5326] 00000008:00010000:3.0:1789760699.784144:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.784151:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760699.784163:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21815296, 21819392) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.784178:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21815296=0x14ce000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.784250:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.784254:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21819392 00000020:00010000:3.0:1789760699.784263:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.784267:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.784270:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.784273:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.784276:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.784284:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.784291:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.784307:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760699.784411:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 21815296<>21819392 00000100:00100000:3.0:1789760699.784460:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760699.786719:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512539264:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760699.847348:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5326, 5326] 00000080:00200000:3.0:1789760699.847364:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.847370:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.847387:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21819392 i_blocks 43015 00000020:00200000:3.0:1789760699.847390:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.847417:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.847453:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21819392, count: 4096 00000080:00200000:3.0:1789760699.847460:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21819392, bytes: 4096 00000080:00200000:3.0:1789760699.847471:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.847480:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.847482:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21819392, 21823488) 0 0 00000080:00200000:3.0:1789760699.847487:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.847503:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5327, 5327] 00020000:00200000:3.0:1789760699.847511:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21819392 chunk: [21819392, 21823488) 21823488, 4096 00020000:00200000:3.0:1789760699.847515:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.847641:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5327, 5327] 00000008:00010000:3.0:1789760699.847659:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.847664:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760699.847676:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21819392, 21823488) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.847687:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21819392=0x14cf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.847761:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.847764:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21823488 00000020:00010000:3.0:1789760699.847772:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.847776:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.847779:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.847782:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.847786:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.847941:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.847948:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.847976:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760699.848091:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107afb80 - object 0x240000400:5 offset 21819392<>21823488 00000080:00200000:3.0:1789760699.918933:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5327, 5327] 00000080:00200000:3.0:1789760699.919870:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.919882:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.919935:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21823488 i_blocks 43015 00000020:00200000:3.0:1789760699.919941:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.919976:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.920013:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21823488, count: 4096 00000080:00200000:3.0:1789760699.920022:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21823488, bytes: 4096 00000080:00200000:3.0:1789760699.923146:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.923167:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.923169:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21823488, 21827584) 0 0 00000080:00200000:3.0:1789760699.923177:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.923198:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5328, 5328] 00020000:00200000:3.0:1789760699.923209:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21823488 chunk: [21823488, 21827584) 21827584, 4096 00020000:00200000:3.0:1789760699.923214:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.923241:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5328, 5328] 00000008:00010000:3.0:1789760699.923261:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.923268:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760699.923280:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21823488, 21827584) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.923296:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21823488=0x14d0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.923371:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.923375:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21827584 00000020:00010000:3.0:1789760699.923383:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.923386:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.923389:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.923392:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.923396:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.923584:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.923592:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.923624:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760699.923990:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 21823488<>21827584 00000080:00200000:3.0:1789760699.993138:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5328, 5328] 00000080:00200000:3.0:1789760699.993156:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760699.993162:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760699.993178:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21827584 i_blocks 43015 00000020:00200000:3.0:1789760699.993182:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760699.993192:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760699.993238:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21827584, count: 4096 00000080:00200000:3.0:1789760699.993245:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21827584, bytes: 4096 00000080:00200000:3.0:1789760699.993258:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760699.993266:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760699.993268:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21827584, 21831680) 0 0 00000080:00200000:3.0:1789760699.993272:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760699.993290:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5329, 5329] 00020000:00200000:3.0:1789760699.993298:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21827584 chunk: [21827584, 21831680) 21831680, 4096 00020000:00200000:3.0:1789760699.993303:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760699.993332:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5329, 5329] 00000008:00010000:3.0:1789760699.993348:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760699.993352:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760699.993361:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21827584, 21831680) append 0 append lockpos 0 00000080:00200000:3.0:1789760699.993373:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21827584=0x14d1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760699.993431:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760699.993434:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21831680 00000020:00010000:3.0:1789760699.993443:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760699.993447:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.993450:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760699.993454:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760699.993457:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760699.993466:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760699.993473:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760699.993490:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760699.993608:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac700 - object 0x240000400:5 offset 21827584<>21831680 00000080:00200000:3.0:1789760700.064722:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5329, 5329] 00000080:00200000:3.0:1789760700.064748:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760700.064756:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760700.064772:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21831680 i_blocks 43015 00000020:00200000:3.0:1789760700.064777:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760700.064796:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760700.064833:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21831680, count: 4096 00000080:00200000:3.0:1789760700.064840:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21831680, bytes: 4096 00000080:00200000:3.0:1789760700.064852:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760700.064861:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760700.064863:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21831680, 21835776) 0 0 00000080:00200000:3.0:1789760700.064867:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760700.064884:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5330, 5330] 00020000:00200000:3.0:1789760700.064893:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21831680 chunk: [21831680, 21835776) 21835776, 4096 00020000:00200000:3.0:1789760700.064936:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760700.064958:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5330, 5330] 00000008:00010000:3.0:1789760700.064976:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760700.064982:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760700.064994:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21831680, 21835776) append 0 append lockpos 0 00000080:00200000:3.0:1789760700.065011:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21831680=0x14d2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760700.065071:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760700.065074:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21835776 00000020:00010000:3.0:1789760700.065093:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760700.065097:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760700.065101:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760700.065104:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760700.065108:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760700.065117:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760700.065236:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760700.065257:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760700.065363:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af480 - object 0x240000400:5 offset 21831680<>21835776 00000080:00200000:3.0:1789760700.167374:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5330, 5330] 00000080:00200000:3.0:1789760700.167393:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760700.167409:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760700.167425:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21835776 i_blocks 43015 00000020:00200000:3.0:1789760700.167430:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760700.167440:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760700.167475:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21835776, count: 4096 00000080:00200000:3.0:1789760700.167483:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21835776, bytes: 4096 00000080:00200000:3.0:1789760700.167691:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760700.167703:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760700.167705:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21835776, 21839872) 0 0 00000080:00200000:3.0:1789760700.167709:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760700.167726:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5331, 5331] 00020000:00200000:3.0:1789760700.167743:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21835776 chunk: [21835776, 21839872) 21839872, 4096 00020000:00200000:3.0:1789760700.167749:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760700.167771:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5331, 5331] 00000008:00010000:3.0:1789760700.167790:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760700.167796:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760700.167807:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21835776, 21839872) append 0 append lockpos 0 00000080:00200000:3.0:1789760700.167821:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21835776=0x14d3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760700.167885:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760700.167887:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21839872 00000020:00010000:3.0:1789760700.167894:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760700.167898:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760700.168023:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760700.168027:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760700.168030:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760700.168041:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760700.168048:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760700.168069:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760700.168343:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae300 - object 0x240000400:5 offset 21835776<>21839872 00000080:00200000:3.0:1789760700.298308:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5331, 5331] 00000080:00200000:3.0:1789760700.298327:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760700.298333:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760700.298350:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21839872 i_blocks 43015 00000020:00200000:3.0:1789760700.298355:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760700.298366:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760700.298411:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21839872, count: 4096 00000080:00200000:3.0:1789760700.298419:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21839872, bytes: 4096 00000080:00200000:3.0:1789760700.298431:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760700.298440:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760700.298442:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21839872, 21843968) 0 0 00000080:00200000:3.0:1789760700.298447:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760700.298466:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5332, 5332] 00020000:00200000:3.0:1789760700.298475:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21839872 chunk: [21839872, 21843968) 21843968, 4096 00020000:00200000:3.0:1789760700.298481:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760700.298722:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5332, 5332] 00000008:00010000:3.0:1789760700.298758:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760700.298764:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760700.298776:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21839872, 21843968) append 0 append lockpos 0 00000080:00200000:3.0:1789760700.298798:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21839872=0x14d4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760700.298860:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760700.298863:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21843968 00000020:00010000:3.0:1789760700.298871:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760700.298875:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760700.298878:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760700.298882:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760700.298885:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760700.298895:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760700.298927:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760700.298945:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760700.299061:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac000 - object 0x240000400:5 offset 21839872<>21843968 00000080:00200000:3.0:1789760700.362387:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5332, 5332] 00000080:00200000:3.0:1789760700.362414:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760700.362421:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760700.362436:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21843968 i_blocks 43015 00000020:00200000:3.0:1789760700.362440:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760700.362451:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760700.362497:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21843968, count: 4096 00000080:00200000:3.0:1789760700.362505:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21843968, bytes: 4096 00000080:00200000:3.0:1789760700.362518:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760700.362527:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760700.362529:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21843968, 21848064) 0 0 00000080:00200000:3.0:1789760700.362534:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760700.362551:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5333, 5333] 00020000:00200000:3.0:1789760700.362560:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21843968 chunk: [21843968, 21848064) 21848064, 4096 00020000:00200000:3.0:1789760700.362564:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760700.362586:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5333, 5333] 00000008:00010000:3.0:1789760700.362609:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760700.362615:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760700.362627:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21843968, 21848064) append 0 append lockpos 0 00000080:00200000:3.0:1789760700.362641:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21843968=0x14d5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760700.362700:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760700.362704:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21848064 00000020:00010000:3.0:1789760700.362713:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760700.362717:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760700.362720:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760700.362723:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760700.362726:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760700.362743:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760700.362751:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760700.362771:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760700.363620:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac380 - object 0x240000400:5 offset 21843968<>21848064 00000100:00100000:1.0:1789760700.609966:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760700.609992:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512540544:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760700.680889:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972655 gen 3 00000008:00010000:1.0:1789760700.680982:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760700.680991:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760700.681003:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760700.681030:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760700.681059:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512540544:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760700.780300:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512540800:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760700.875531:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760700.875561:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512540928:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760700.941074:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760700.941091:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760700.941105:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760700.941133:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760700.941166:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512540928:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760701.021680:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5341, 5341] 00000080:00200000:3.0:1789760701.021697:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760701.021702:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760701.021720:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21880832 i_blocks 43015 00000020:00200000:3.0:1789760701.021724:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760701.021747:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760701.021786:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21880832, count: 4096 00000080:00200000:3.0:1789760701.021793:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21880832, bytes: 4096 00000080:00200000:3.0:1789760701.021803:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760701.021812:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760701.021815:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21880832, 21884928) 0 0 00000080:00200000:3.0:1789760701.021819:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760701.021838:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5342, 5342] 00020000:00200000:3.0:1789760701.021848:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21880832 chunk: [21880832, 21884928) 21884928, 4096 00020000:00200000:3.0:1789760701.021853:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760701.021873:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5342, 5342] 00000008:00010000:3.0:1789760701.021892:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760701.021898:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760701.021950:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21880832, 21884928) append 0 append lockpos 0 00000080:00200000:3.0:1789760701.021967:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21880832=0x14de000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760701.022034:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760701.022037:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21884928 00000020:00010000:3.0:1789760701.022045:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760701.022048:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.022052:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.022056:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760701.022059:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760701.022067:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760701.022074:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760701.022090:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760701.022203:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aea00 - object 0x240000400:5 offset 21880832<>21884928 00000080:00200000:3.0:1789760701.054551:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5342, 5342] 00000080:00200000:3.0:1789760701.054703:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760701.054710:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760701.054735:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21884928 i_blocks 43015 00000020:00200000:3.0:1789760701.054740:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760701.054750:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760701.054787:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21884928, count: 4096 00000080:00200000:3.0:1789760701.054794:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21884928, bytes: 4096 00000080:00200000:3.0:1789760701.054805:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760701.054814:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760701.054817:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21884928, 21889024) 0 0 00000080:00200000:3.0:1789760701.054822:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760701.054838:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5343, 5343] 00020000:00200000:3.0:1789760701.054846:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21884928 chunk: [21884928, 21889024) 21889024, 4096 00020000:00200000:3.0:1789760701.054851:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760701.054871:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5343, 5343] 00000008:00010000:3.0:1789760701.054891:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760701.054897:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760701.055593:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21884928, 21889024) append 0 append lockpos 0 00000080:00200000:3.0:1789760701.055609:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21884928=0x14df000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760701.055756:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760701.055760:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21889024 00000020:00010000:3.0:1789760701.055836:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760701.055841:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.055845:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.055849:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760701.055853:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760701.055862:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760701.055870:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760701.055897:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760701.056220:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af100 - object 0x240000400:5 offset 21884928<>21889024 00000080:00200000:3.0:1789760701.099684:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5343, 5343] 00000080:00200000:3.0:1789760701.099707:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760701.099714:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760701.099741:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21889024 i_blocks 43015 00000020:00200000:3.0:1789760701.099746:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760701.099757:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760701.099802:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21889024, count: 4096 00000080:00200000:3.0:1789760701.099811:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21889024, bytes: 4096 00000080:00200000:3.0:1789760701.099821:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760701.099831:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760701.099834:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21889024, 21893120) 0 0 00000080:00200000:3.0:1789760701.099838:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760701.099857:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5344, 5344] 00020000:00200000:3.0:1789760701.099866:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21889024 chunk: [21889024, 21893120) 21893120, 4096 00020000:00200000:3.0:1789760701.099871:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760701.099894:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5344, 5344] 00000008:00010000:3.0:1789760701.100641:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760701.100649:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760701.100662:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21889024, 21893120) append 0 append lockpos 0 00000080:00200000:3.0:1789760701.100677:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21889024=0x14e0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760701.100754:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760701.100758:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21893120 00000020:00010000:3.0:1789760701.100826:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760701.100831:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.100835:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.100838:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760701.100842:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760701.100851:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760701.100859:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760701.100881:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760701.101229:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad180 - object 0x240000400:5 offset 21889024<>21893120 00000080:00200000:3.0:1789760701.140641:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5344, 5344] 00000080:00200000:3.0:1789760701.140652:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760701.140657:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760701.140667:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21893120 i_blocks 43015 00000020:00200000:3.0:1789760701.140671:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760701.140677:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760701.140701:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21893120, count: 4096 00000080:00200000:3.0:1789760701.140707:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21893120, bytes: 4096 00000080:00200000:3.0:1789760701.140716:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760701.140722:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760701.140734:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21893120, 21897216) 0 0 00000080:00200000:3.0:1789760701.140738:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760701.140751:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5345, 5345] 00020000:00200000:3.0:1789760701.140816:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21893120 chunk: [21893120, 21897216) 21897216, 4096 00020000:00200000:3.0:1789760701.140822:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760701.140838:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5345, 5345] 00000008:00010000:3.0:1789760701.140861:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760701.140866:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760701.140875:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21893120, 21897216) append 0 append lockpos 0 00000080:00200000:3.0:1789760701.140885:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21893120=0x14e1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760701.140961:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760701.140964:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21897216 00000020:00010000:3.0:1789760701.140972:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760701.140976:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.140979:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.140982:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760701.140986:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760701.140994:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760701.141001:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760701.141016:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760701.141091:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adf80 - object 0x240000400:5 offset 21893120<>21897216 00000100:00100000:3.0:1789760701.141131:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760701.141145:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512541696:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760701.196940:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5345, 5345] 00000080:00200000:3.0:1789760701.196956:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760701.196962:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760701.196976:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21897216 i_blocks 43015 00000020:00200000:3.0:1789760701.196980:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760701.196990:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760701.197024:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21897216, count: 4096 00000080:00200000:3.0:1789760701.197032:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21897216, bytes: 4096 00000080:00200000:3.0:1789760701.197043:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760701.197052:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760701.197055:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21897216, 21901312) 0 0 00000080:00200000:3.0:1789760701.197059:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760701.197087:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5346, 5346] 00020000:00200000:3.0:1789760701.197095:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21897216 chunk: [21897216, 21901312) 21901312, 4096 00020000:00200000:3.0:1789760701.197099:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760701.197232:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5346, 5346] 00000008:00010000:3.0:1789760701.197251:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760701.197259:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760701.197271:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21897216, 21901312) append 0 append lockpos 0 00000080:00200000:3.0:1789760701.197285:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21897216=0x14e2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760701.197346:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760701.197349:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21901312 00000020:00010000:3.0:1789760701.197358:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760701.197362:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.197365:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.197369:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760701.197371:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760701.197380:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760701.197387:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760701.197404:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760701.197512:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad880 - object 0x240000400:5 offset 21897216<>21901312 00000080:00200000:3.0:1789760701.249482:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5346, 5346] 00000080:00200000:3.0:1789760701.249499:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760701.249505:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760701.249521:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21901312 i_blocks 43015 00000020:00200000:3.0:1789760701.249526:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760701.249536:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760701.249568:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21901312, count: 4096 00000080:00200000:3.0:1789760701.249574:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21901312, bytes: 4096 00000080:00200000:3.0:1789760701.249583:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760701.249592:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760701.249594:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21901312, 21905408) 0 0 00000080:00200000:3.0:1789760701.249599:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760701.249613:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5347, 5347] 00020000:00200000:3.0:1789760701.249623:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21901312 chunk: [21901312, 21905408) 21905408, 4096 00020000:00200000:3.0:1789760701.249627:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760701.249646:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5347, 5347] 00000008:00010000:3.0:1789760701.249665:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760701.249670:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760701.249682:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21901312, 21905408) append 0 append lockpos 0 00000080:00200000:3.0:1789760701.249694:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21901312=0x14e3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760701.249758:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760701.249761:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21905408 00000020:00010000:3.0:1789760701.249769:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760701.249772:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.249775:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.249779:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760701.249782:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760701.249791:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760701.249798:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760701.249815:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760701.249964:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 21901312<>21905408 00000100:00100000:3.0:1789760701.284987:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972666 gen 3 00000008:00010000:3.0:1789760701.285036:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:3.0:1789760701.285045:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760701.285057:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760701.285071:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:3.0:1789760701.285097:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512541952:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760701.288656:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760701.288686:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512542080:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760701.361551:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760701.361569:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760701.361582:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760701.362009:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760701.362042:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512542080:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760701.364969:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512542208:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760701.423454:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972668 gen 3 00000008:00010000:1.0:1789760701.423519:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760701.423528:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760701.423538:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760701.424292:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760701.424328:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512542208:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760701.464287:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972669 gen 3 00000008:00010000:1.0:1789760701.464342:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760701.464352:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760701.464363:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760701.464388:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760701.464431:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512542336:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760701.517864:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5351, 5351] 00000080:00200000:1.0:1789760701.517886:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760701.517892:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760701.517938:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21921792 i_blocks 43015 00000020:00200000:1.0:1789760701.517943:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760701.517954:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760701.517986:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21921792, count: 4096 00000080:00200000:1.0:1789760701.517994:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21921792, bytes: 4096 00000080:00200000:1.0:1789760701.518004:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760701.518014:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760701.518016:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21921792, 21925888) 0 0 00000080:00200000:1.0:1789760701.518021:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760701.518039:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5352, 5352] 00020000:00200000:1.0:1789760701.518048:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21921792 chunk: [21921792, 21925888) 21925888, 4096 00020000:00200000:1.0:1789760701.518052:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760701.518072:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5352, 5352] 00000008:00010000:1.0:1789760701.518090:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760701.518096:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760701.518108:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21921792, 21925888) append 0 append lockpos 0 00000080:00200000:1.0:1789760701.518121:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21921792=0x14e8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760701.518184:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760701.518187:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21925888 00000020:00010000:1.0:1789760701.518196:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:1.0:1789760701.518200:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760701.518203:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760701.518206:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760701.518209:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760701.518217:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760701.518223:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760701.518240:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f429c0 00000008:00100000:1.0:1789760701.518342:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 21921792<>21925888 00000080:00200000:3.0:1789760701.559506:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5352, 5352] 00000080:00200000:3.0:1789760701.559523:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760701.559529:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760701.559544:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21925888 i_blocks 43015 00000020:00200000:3.0:1789760701.559548:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760701.559558:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760701.559594:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21925888, count: 4096 00000080:00200000:3.0:1789760701.559600:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21925888, bytes: 4096 00000080:00200000:3.0:1789760701.559612:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760701.559622:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760701.559624:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21925888, 21929984) 0 0 00000080:00200000:3.0:1789760701.559629:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760701.559646:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5353, 5353] 00020000:00200000:3.0:1789760701.559655:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21925888 chunk: [21925888, 21929984) 21929984, 4096 00020000:00200000:3.0:1789760701.559660:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760701.559680:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5353, 5353] 00000008:00010000:3.0:1789760701.559698:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760701.559703:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760701.559715:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21925888, 21929984) append 0 append lockpos 0 00000080:00200000:3.0:1789760701.559741:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21925888=0x14e9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760701.560565:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760701.560573:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21929984 00000020:00010000:3.0:1789760701.560586:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760701.560590:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.560594:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.560598:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760701.560601:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760701.560612:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760701.560622:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760701.560653:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760701.560782:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 21925888<>21929984 00000080:00200000:3.0:1789760701.599559:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5353, 5353] 00000080:00200000:3.0:1789760701.599575:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760701.599582:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760701.599597:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21929984 i_blocks 43015 00000020:00200000:3.0:1789760701.599601:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760701.599610:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760701.599644:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21929984, count: 4096 00000080:00200000:3.0:1789760701.599651:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21929984, bytes: 4096 00000080:00200000:3.0:1789760701.599661:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760701.599671:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760701.599673:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21929984, 21934080) 0 0 00000080:00200000:3.0:1789760701.599677:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760701.599695:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5354, 5354] 00020000:00200000:3.0:1789760701.599703:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21929984 chunk: [21929984, 21934080) 21934080, 4096 00020000:00200000:3.0:1789760701.599708:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760701.599737:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5354, 5354] 00000008:00010000:3.0:1789760701.599754:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760701.599761:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760701.599772:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21929984, 21934080) append 0 append lockpos 0 00000080:00200000:3.0:1789760701.599785:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21929984=0x14ea000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760701.599939:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760701.599942:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21934080 00000020:00010000:3.0:1789760701.599952:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760701.599955:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.599958:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.599962:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760701.599965:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760701.599973:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760701.599979:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760701.599996:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760701.600096:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae680 - object 0x240000400:5 offset 21929984<>21934080 00000080:00200000:3.0:1789760701.647244:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5354, 5354] 00000080:00200000:3.0:1789760701.647261:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760701.647268:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760701.647285:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21934080 i_blocks 43015 00000020:00200000:3.0:1789760701.647288:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760701.647298:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760701.647334:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21934080, count: 4096 00000080:00200000:3.0:1789760701.647342:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21934080, bytes: 4096 00000080:00200000:3.0:1789760701.647352:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760701.647360:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760701.647363:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21934080, 21938176) 0 0 00000080:00200000:3.0:1789760701.647367:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760701.647383:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5355, 5355] 00020000:00200000:3.0:1789760701.647403:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21934080 chunk: [21934080, 21938176) 21938176, 4096 00020000:00200000:3.0:1789760701.647410:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760701.647431:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5355, 5355] 00000008:00010000:3.0:1789760701.647449:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760701.647455:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760701.647468:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21934080, 21938176) append 0 append lockpos 0 00000080:00200000:3.0:1789760701.647482:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21934080=0x14eb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760701.647542:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760701.647545:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21938176 00000020:00010000:3.0:1789760701.647552:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760701.647555:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.647558:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.647562:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760701.647565:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760701.647574:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760701.647581:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760701.647600:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760701.647980:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 21934080<>21938176 00000080:00200000:3.0:1789760701.745969:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5355, 5355] 00000080:00200000:3.0:1789760701.745987:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760701.745993:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760701.746008:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21938176 i_blocks 43015 00000020:00200000:3.0:1789760701.746012:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760701.746022:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760701.746057:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21938176, count: 4096 00000080:00200000:3.0:1789760701.746065:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21938176, bytes: 4096 00000080:00200000:3.0:1789760701.746075:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760701.746084:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760701.746086:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21938176, 21942272) 0 0 00000080:00200000:3.0:1789760701.746091:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760701.746107:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5356, 5356] 00020000:00200000:3.0:1789760701.746128:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21938176 chunk: [21938176, 21942272) 21942272, 4096 00020000:00200000:3.0:1789760701.746134:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760701.746155:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5356, 5356] 00000008:00010000:3.0:1789760701.746173:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760701.746179:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760701.746191:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21938176, 21942272) append 0 append lockpos 0 00000080:00200000:3.0:1789760701.746205:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21938176=0x14ec000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760701.746264:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760701.746267:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21942272 00000020:00010000:3.0:1789760701.746281:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760701.746284:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.746288:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.746292:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760701.746295:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760701.746303:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760701.746314:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760701.746331:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760701.746446:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af800 - object 0x240000400:5 offset 21938176<>21942272 00000080:00200000:3.0:1789760701.802797:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5356, 5356] 00000080:00200000:3.0:1789760701.802814:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760701.802821:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760701.802836:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21942272 i_blocks 43015 00000020:00200000:3.0:1789760701.802840:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760701.802849:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760701.802883:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21942272, count: 4096 00000080:00200000:3.0:1789760701.802891:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21942272, bytes: 4096 00000080:00200000:3.0:1789760701.803053:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760701.803064:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760701.803067:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21942272, 21946368) 0 0 00000080:00200000:3.0:1789760701.803073:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760701.803090:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5357, 5357] 00020000:00200000:3.0:1789760701.803098:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21942272 chunk: [21942272, 21946368) 21946368, 4096 00020000:00200000:3.0:1789760701.803104:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760701.803186:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5357, 5357] 00000008:00010000:3.0:1789760701.803208:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760701.803214:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760701.803227:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21942272, 21946368) append 0 append lockpos 0 00000080:00200000:3.0:1789760701.803241:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21942272=0x14ed000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760701.803302:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760701.803305:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21946368 00000020:00010000:3.0:1789760701.803316:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760701.803319:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.803323:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.803326:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760701.803329:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760701.803339:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760701.803347:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760701.803368:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760701.803534:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af800 - object 0x240000400:5 offset 21942272<>21946368 00000080:00200000:3.0:1789760701.869223:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5357, 5357] 00000080:00200000:3.0:1789760701.869305:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760701.869312:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760701.869332:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21946368 i_blocks 43015 00000020:00200000:3.0:1789760701.869336:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760701.869348:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760701.869383:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21946368, count: 4096 00000080:00200000:3.0:1789760701.869391:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21946368, bytes: 4096 00000080:00200000:3.0:1789760701.869404:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760701.869413:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760701.869415:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21946368, 21950464) 0 0 00000080:00200000:3.0:1789760701.869419:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760701.869436:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5358, 5358] 00020000:00200000:3.0:1789760701.869443:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21946368 chunk: [21946368, 21950464) 21950464, 4096 00020000:00200000:3.0:1789760701.869450:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760701.869470:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5358, 5358] 00000008:00010000:3.0:1789760701.869488:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760701.869494:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760701.869507:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21946368, 21950464) append 0 append lockpos 0 00000080:00200000:3.0:1789760701.869523:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21946368=0x14ee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760701.870171:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760701.870175:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21950464 00000020:00010000:3.0:1789760701.870186:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760701.870190:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.870193:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.870197:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760701.870199:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760701.870210:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760701.870218:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760701.870303:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760701.870414:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 21946368<>21950464 00000080:00200000:3.0:1789760701.954713:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5358, 5358] 00000080:00200000:3.0:1789760701.954743:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760701.954749:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760701.954764:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21950464 i_blocks 43015 00000020:00200000:3.0:1789760701.954767:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760701.954777:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760701.954811:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21950464, count: 4096 00000080:00200000:3.0:1789760701.954818:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21950464, bytes: 4096 00000080:00200000:3.0:1789760701.954827:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760701.954836:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760701.954839:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21950464, 21954560) 0 0 00000080:00200000:3.0:1789760701.954843:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760701.954859:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5359, 5359] 00020000:00200000:3.0:1789760701.954867:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21950464 chunk: [21950464, 21954560) 21954560, 4096 00020000:00200000:3.0:1789760701.954871:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760701.954889:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5359, 5359] 00000008:00010000:3.0:1789760701.954925:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760701.954931:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760701.954941:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21950464, 21954560) append 0 append lockpos 0 00000080:00200000:3.0:1789760701.954954:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21950464=0x14ef000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760701.955004:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760701.955006:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21954560 00000020:00010000:3.0:1789760701.955014:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760701.955017:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.955020:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760701.955024:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760701.955027:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760701.955034:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760701.955040:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760701.955054:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760701.955344:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae680 - object 0x240000400:5 offset 21950464<>21954560 00000080:00200000:3.0:1789760702.004481:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5359, 5359] 00000080:00200000:3.0:1789760702.004498:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760702.004505:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760702.004522:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21954560 i_blocks 43015 00000020:00200000:3.0:1789760702.004525:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760702.004535:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760702.004571:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21954560, count: 4096 00000080:00200000:3.0:1789760702.004579:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21954560, bytes: 4096 00000080:00200000:3.0:1789760702.004589:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760702.004598:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760702.004600:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21954560, 21958656) 0 0 00000080:00200000:3.0:1789760702.004605:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760702.004622:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5360, 5360] 00020000:00200000:3.0:1789760702.004632:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21954560 chunk: [21954560, 21958656) 21958656, 4096 00020000:00200000:3.0:1789760702.004637:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760702.004656:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5360, 5360] 00000008:00010000:3.0:1789760702.004673:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760702.004680:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760702.004691:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21954560, 21958656) append 0 append lockpos 0 00000080:00200000:3.0:1789760702.004707:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21954560=0x14f0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760702.004786:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760702.004789:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21958656 00000020:00010000:3.0:1789760702.004796:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760702.004800:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760702.004803:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760702.004807:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760702.004809:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760702.004818:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760702.004825:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760702.004844:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760702.004974:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 21954560<>21958656 00000080:00200000:3.0:1789760702.072894:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5360, 5360] 00000080:00200000:3.0:1789760702.072940:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760702.072947:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760702.072962:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21958656 i_blocks 43015 00000020:00200000:3.0:1789760702.072965:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760702.072975:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760702.073009:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21958656, count: 4096 00000080:00200000:3.0:1789760702.073018:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21958656, bytes: 4096 00000080:00200000:3.0:1789760702.073028:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760702.073037:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760702.073039:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21958656, 21962752) 0 0 00000080:00200000:3.0:1789760702.073044:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760702.073062:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5361, 5361] 00020000:00200000:3.0:1789760702.073071:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21958656 chunk: [21958656, 21962752) 21962752, 4096 00020000:00200000:3.0:1789760702.073076:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760702.073097:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5361, 5361] 00000008:00010000:3.0:1789760702.073176:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760702.073182:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760702.073195:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21958656, 21962752) append 0 append lockpos 0 00000080:00200000:3.0:1789760702.073209:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21958656=0x14f1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760702.073268:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760702.073272:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21962752 00000020:00010000:3.0:1789760702.073282:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760702.073286:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760702.073289:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760702.073294:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760702.073297:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760702.073305:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760702.073312:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760702.073330:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760702.073453:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad880 - object 0x240000400:5 offset 21958656<>21962752 00000080:00200000:3.0:1789760702.111401:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5361, 5361] 00000080:00200000:3.0:1789760702.111417:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760702.111423:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760702.111439:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21962752 i_blocks 43015 00000020:00200000:3.0:1789760702.111442:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760702.111451:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760702.111484:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21962752, count: 4096 00000080:00200000:3.0:1789760702.111491:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21962752, bytes: 4096 00000080:00200000:3.0:1789760702.111501:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760702.111510:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760702.111512:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21962752, 21966848) 0 0 00000080:00200000:3.0:1789760702.111517:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760702.111542:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5362, 5362] 00020000:00200000:3.0:1789760702.111552:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21962752 chunk: [21962752, 21966848) 21966848, 4096 00020000:00200000:3.0:1789760702.111558:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760702.111578:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5362, 5362] 00000008:00010000:3.0:1789760702.111597:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760702.111604:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760702.111617:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21962752, 21966848) append 0 append lockpos 0 00000080:00200000:3.0:1789760702.111631:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21962752=0x14f2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760702.111694:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760702.111696:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21966848 00000020:00010000:3.0:1789760702.111706:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_release():245 00000020:00010000:3.0:1789760702.111709:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760702.111712:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c998/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760702.111716:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0189278a8 (ffff8dc00877cd88/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760702.111730:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc00877cd88/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760702.111739:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760702.111746:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760702.111762:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760702.111869:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adf80 - object 0x240000400:5 offset 21962752<>21966848 00000080:00200000:3.0:1789760702.150950:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5362, 5362] 00000080:00200000:3.0:1789760702.150973:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760702.150981:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760702.150999:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21966848 i_blocks 43015 00000020:00200000:3.0:1789760702.151003:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760702.151014:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200000:3.0:1789760702.195142:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5363, 5363] 00000080:00200000:3.0:1789760702.195159:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760702.195165:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760702.195182:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21970944 i_blocks 43015 00000020:00200000:3.0:1789760702.195187:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760702.195198:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760702.195233:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21970944, count: 4096 00000080:00200000:3.0:1789760702.195242:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21970944, bytes: 4096 00000080:00200000:3.0:1789760702.195253:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760702.195262:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760702.195264:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21970944, 21975040) 0 0 00000080:00200000:3.0:1789760702.195269:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760702.195286:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5364, 5364] 00020000:00200000:3.0:1789760702.195296:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21970944 chunk: [21970944, 21975040) 21975040, 4096 00020000:00200000:3.0:1789760702.195301:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760702.195321:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5364, 5364] 00000008:00010000:3.0:1789760702.195342:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760702.195348:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760702.195359:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21970944, 21975040) append 0 append lockpos 0 00000080:00200000:3.0:1789760702.195376:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21970944=0x14f4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760702.195793:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760702.195797:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21975040 00000020:00010000:3.0:1789760702.195806:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:3.0:1789760702.195810:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760702.195814:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760702.195818:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760702.195821:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760702.195830:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760702.195837:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760702.195859:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760702.196029:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad180 - object 0x240000400:5 offset 21970944<>21975040 00010000:00010000:3.0:1789760702.237364:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:1.0:1789760702.440566:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760702.440577:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760702.440587:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760702.440611:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760702.440638:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512544512:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760702.441584:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512544640:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760702.540555:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5368, 5368] 00000080:00200000:1.0:1789760702.540567:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760702.540573:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760702.540585:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21991424 i_blocks 43015 00000020:00200000:1.0:1789760702.540588:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760702.540596:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760702.540623:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21991424, count: 4096 00000080:00200000:1.0:1789760702.540629:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21991424, bytes: 4096 00000080:00200000:1.0:1789760702.540638:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760702.540646:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760702.540648:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21991424, 21995520) 0 0 00000080:00200000:1.0:1789760702.540652:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760702.540666:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5369, 5369] 00020000:00200000:1.0:1789760702.540674:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21991424 chunk: [21991424, 21995520) 21995520, 4096 00020000:00200000:1.0:1789760702.540678:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760702.540694:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5369, 5369] 00000008:00010000:1.0:1789760702.540708:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760702.540712:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760702.540731:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21991424, 21995520) append 0 append lockpos 0 00000080:00200000:1.0:1789760702.540742:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21991424=0x14f9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760702.540792:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760702.540868:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 21995520 00000020:00010000:1.0:1789760702.540876:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:1.0:1789760702.540879:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760702.540882:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760702.540885:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760702.540888:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760702.540894:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760702.540943:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760702.540959:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760702.541036:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95c00 - object 0x240000400:5 offset 21991424<>21995520 00000100:00100000:1.0:1789760702.541068:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760702.541082:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512544768:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760702.629763:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972688 gen 3 00000008:00010000:1.0:1789760702.629826:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760702.629832:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760702.629841:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760702.629861:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760702.629884:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512544768:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760702.719017:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5370, 5370] 00000080:00200000:1.0:1789760702.719030:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760702.719035:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760702.719045:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 21999616 i_blocks 43015 00000020:00200000:1.0:1789760702.719048:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760702.719057:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760702.719084:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 21999616, count: 4096 00000080:00200000:1.0:1789760702.719090:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 21999616, bytes: 4096 00000080:00200000:1.0:1789760702.719098:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760702.719167:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760702.719169:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [21999616, 22003712) 0 0 00000080:00200000:1.0:1789760702.719173:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760702.719185:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5371, 5371] 00020000:00200000:1.0:1789760702.719191:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 21999616 chunk: [21999616, 22003712) 22003712, 4096 00020000:00200000:1.0:1789760702.719195:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760702.719210:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5371, 5371] 00000008:00010000:1.0:1789760702.719222:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760702.719227:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760702.719235:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [21999616, 22003712) append 0 append lockpos 0 00000080:00200000:1.0:1789760702.719247:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=21999616=0x14fb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760702.719292:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760702.719294:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22003712 00000020:00010000:1.0:1789760702.719301:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:1.0:1789760702.719304:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760702.719306:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760702.719309:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760702.719312:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760702.719318:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760702.719323:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760702.719338:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760702.719582:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95c00 - object 0x240000400:5 offset 21999616<>22003712 00000080:00200000:1.0:1789760702.843668:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5371, 5371] 00000080:00200000:1.0:1789760702.843685:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760702.843691:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760702.843708:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22003712 i_blocks 43015 00000020:00200000:1.0:1789760702.843791:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760702.843803:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760702.843841:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22003712, count: 4096 00000080:00200000:1.0:1789760702.843850:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22003712, bytes: 4096 00000080:00200000:1.0:1789760702.843863:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760702.843873:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760702.843875:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22003712, 22007808) 0 0 00000080:00200000:1.0:1789760702.843880:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760702.843926:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5372, 5372] 00020000:00200000:1.0:1789760702.843936:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22003712 chunk: [22003712, 22007808) 22007808, 4096 00020000:00200000:1.0:1789760702.843943:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760702.843964:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5372, 5372] 00000008:00010000:1.0:1789760702.843984:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760702.843989:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760702.844001:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22003712, 22007808) append 0 append lockpos 0 00000080:00200000:1.0:1789760702.844014:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22003712=0x14fc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760702.844091:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760702.844094:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22007808 00000020:00010000:1.0:1789760702.844225:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:1.0:1789760702.844230:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760702.844233:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760702.844237:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760702.844239:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760702.844249:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760702.844256:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760702.844277:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760702.844404:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 22003712<>22007808 00000080:00200000:1.0:1789760702.917831:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5372, 5372] 00000080:00200000:1.0:1789760702.917848:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760702.917854:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760702.917868:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22007808 i_blocks 43015 00000020:00200000:1.0:1789760702.917872:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760702.917883:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760702.918504:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22007808, count: 4096 00000080:00200000:1.0:1789760702.918515:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22007808, bytes: 4096 00000080:00200000:1.0:1789760702.918528:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760702.918538:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760702.918540:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22007808, 22011904) 0 0 00000080:00200000:1.0:1789760702.918545:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760702.918563:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5373, 5373] 00020000:00200000:1.0:1789760702.918572:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22007808 chunk: [22007808, 22011904) 22011904, 4096 00020000:00200000:1.0:1789760702.918577:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760702.918598:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5373, 5373] 00000008:00010000:1.0:1789760702.918616:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760702.918621:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760702.918634:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22007808, 22011904) append 0 append lockpos 0 00000080:00200000:1.0:1789760702.918649:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22007808=0x14fd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760702.918702:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760702.918704:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22011904 00000020:00010000:1.0:1789760702.918724:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:1.0:1789760702.918728:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760702.918731:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760702.918735:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760702.918737:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760702.918746:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760702.918752:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760702.918770:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f429c0 00000008:00100000:1.0:1789760702.919176:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 22007808<>22011904 00000080:00200000:1.0:1789760702.972219:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5373, 5373] 00000080:00200000:1.0:1789760702.972239:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760702.972245:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760702.972262:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22011904 i_blocks 43015 00000020:00200000:1.0:1789760702.972265:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760702.972277:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760702.972313:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22011904, count: 4096 00000080:00200000:1.0:1789760702.972321:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22011904, bytes: 4096 00000080:00200000:1.0:1789760702.972333:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760702.972342:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760702.972344:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22011904, 22016000) 0 0 00000080:00200000:1.0:1789760702.972349:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760702.972365:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5374, 5374] 00020000:00200000:1.0:1789760702.972373:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22011904 chunk: [22011904, 22016000) 22016000, 4096 00020000:00200000:1.0:1789760702.972378:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760702.972588:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5374, 5374] 00000008:00010000:1.0:1789760702.972609:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760702.972614:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760702.972627:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22011904, 22016000) append 0 append lockpos 0 00000080:00200000:1.0:1789760702.972642:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22011904=0x14fe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760702.972725:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760702.972729:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22016000 00000020:00010000:1.0:1789760702.972739:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:1.0:1789760702.972743:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760702.972747:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760702.972751:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760702.972754:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760702.972764:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760702.972771:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760702.972797:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42900 00000008:00100000:1.0:1789760702.972983:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94700 - object 0x240000400:5 offset 22011904<>22016000 00000100:00100000:1.0:1789760702.973037:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512545408:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760703.042197:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972693 gen 3 00000008:00010000:1.0:1789760703.042249:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42900 00000020:00200000:1.0:1789760703.042258:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760703.042270:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760703.042292:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42900 00000100:00100000:1.0:1789760703.042322:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512545408:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760703.123423:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512545664:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760703.163878:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512545792:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760703.244336:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760703.244353:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760703.244367:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760703.244393:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760703.244426:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512545792:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760703.245263:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512545920:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760703.317018:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972697 gen 3 00000008:00010000:1.0:1789760703.317084:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760703.317092:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760703.317103:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760703.317127:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760703.317156:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512545920:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760703.317993:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760703.318010:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512546432:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760703.356705:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972698 gen 3 00000008:00010000:1.0:1789760703.356768:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760703.356777:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760703.356993:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760703.357017:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760703.357046:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512546432:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760703.359050:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512546560:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760703.398549:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972699 gen 3 00000008:00010000:1.0:1789760703.398604:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760703.398613:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760703.398626:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760703.398701:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760703.398754:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512546560:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760703.399540:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512546688:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760703.484763:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512546816:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760703.669686:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5382, 5382] 00000080:00200000:1.0:1789760703.669703:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760703.669720:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760703.669736:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22048768 i_blocks 45063 00000020:00200000:1.0:1789760703.669740:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760703.669751:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760703.669851:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22048768, count: 4096 00000080:00200000:1.0:1789760703.669861:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22048768, bytes: 4096 00000080:00200000:1.0:1789760703.669874:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760703.669884:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760703.669886:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22048768, 22052864) 0 0 00000080:00200000:1.0:1789760703.669891:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760703.669934:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5383, 5383] 00020000:00200000:1.0:1789760703.669942:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22048768 chunk: [22048768, 22052864) 22052864, 4096 00020000:00200000:1.0:1789760703.669947:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760703.669969:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5383, 5383] 00000008:00010000:1.0:1789760703.669987:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760703.669994:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760703.670007:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22048768, 22052864) append 0 append lockpos 0 00000080:00200000:1.0:1789760703.670022:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22048768=0x1507000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760703.670081:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760703.670085:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22052864 00000020:00010000:1.0:1789760703.670093:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:1.0:1789760703.670223:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760703.670228:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760703.670232:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760703.670235:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760703.670244:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760703.670251:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760703.670270:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42900 00000008:00100000:1.0:1789760703.670375:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94700 - object 0x240000400:5 offset 22048768<>22052864 00000080:00200000:3.0:1789760703.745327:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5383, 5383] 00000080:00200000:3.0:1789760703.745347:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760703.745354:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760703.745370:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22052864 i_blocks 45063 00000020:00200000:3.0:1789760703.745374:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760703.745513:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760703.745555:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22052864, count: 4096 00000080:00200000:3.0:1789760703.745565:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22052864, bytes: 4096 00000080:00200000:3.0:1789760703.745580:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760703.745591:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760703.745594:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22052864, 22056960) 0 0 00000080:00200000:3.0:1789760703.745600:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760703.745617:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5384, 5384] 00020000:00200000:3.0:1789760703.745626:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22052864 chunk: [22052864, 22056960) 22056960, 4096 00020000:00200000:3.0:1789760703.745631:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760703.745655:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5384, 5384] 00000008:00010000:3.0:1789760703.745681:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760703.745688:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760703.745702:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22052864, 22056960) append 0 append lockpos 0 00000080:00200000:3.0:1789760703.745728:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22052864=0x1508000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760703.745788:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760703.745791:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22056960 00000020:00010000:3.0:1789760703.745800:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:3.0:1789760703.745804:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760703.746522:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760703.746527:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760703.746530:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760703.746544:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760703.746554:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760703.821851:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5384, 5384] 00000080:00200000:3.0:1789760703.821869:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760703.821875:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760703.821890:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22056960 i_blocks 45063 00000020:00200000:3.0:1789760703.821894:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760703.821946:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760703.821981:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22056960, count: 4096 00000080:00200000:3.0:1789760703.821989:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22056960, bytes: 4096 00000080:00200000:3.0:1789760703.821999:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760703.822007:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760703.822010:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22056960, 22061056) 0 0 00000080:00200000:3.0:1789760703.822014:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760703.822032:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5385, 5385] 00020000:00200000:3.0:1789760703.822041:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22056960 chunk: [22056960, 22061056) 22061056, 4096 00020000:00200000:3.0:1789760703.822046:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760703.822067:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5385, 5385] 00000008:00010000:3.0:1789760703.822085:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760703.822092:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760703.822208:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22056960, 22061056) append 0 append lockpos 0 00000080:00200000:3.0:1789760703.822224:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22056960=0x1509000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760703.822285:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760703.822288:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22061056 00000020:00010000:3.0:1789760703.822297:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:3.0:1789760703.822300:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760703.822303:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760703.822306:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760703.822308:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760703.822318:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760703.822325:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760703.822343:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760703.822453:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac000 - object 0x240000400:5 offset 22056960<>22061056 00000080:00200000:3.0:1789760703.979470:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5385, 5385] 00000080:00200000:3.0:1789760703.979490:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760703.979497:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760703.979513:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22061056 i_blocks 45063 00000020:00200000:3.0:1789760703.979518:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760703.979529:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760703.979567:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22061056, count: 4096 00000080:00200000:3.0:1789760703.979574:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22061056, bytes: 4096 00000080:00200000:3.0:1789760703.979585:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760703.979595:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760703.979598:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22061056, 22065152) 0 0 00000080:00200000:3.0:1789760703.979603:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760703.979620:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5386, 5386] 00020000:00200000:3.0:1789760703.979629:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22061056 chunk: [22061056, 22065152) 22065152, 4096 00020000:00200000:3.0:1789760703.979633:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760703.979651:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5386, 5386] 00000008:00010000:3.0:1789760703.979670:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760703.979676:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760703.979687:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22061056, 22065152) append 0 append lockpos 0 00000080:00200000:3.0:1789760703.979702:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22061056=0x150a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760703.979774:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760703.979777:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22065152 00000020:00010000:3.0:1789760703.979786:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:3.0:1789760703.979790:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760703.979793:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760703.979797:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760703.979800:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760703.979808:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760703.979815:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760703.979833:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760703.979987:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae300 - object 0x240000400:5 offset 22061056<>22065152 00000080:00200000:3.0:1789760704.115251:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5386, 5386] 00000080:00200000:3.0:1789760704.115271:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760704.115277:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760704.115291:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22065152 i_blocks 45063 00000020:00200000:3.0:1789760704.115296:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760704.115318:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760704.115354:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22065152, count: 4096 00000080:00200000:3.0:1789760704.115363:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22065152, bytes: 4096 00000080:00200000:3.0:1789760704.115486:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760704.115497:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760704.115499:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22065152, 22069248) 0 0 00000080:00200000:3.0:1789760704.115504:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760704.115521:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5387, 5387] 00020000:00200000:3.0:1789760704.115529:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22065152 chunk: [22065152, 22069248) 22069248, 4096 00020000:00200000:3.0:1789760704.115534:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760704.115556:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5387, 5387] 00000008:00010000:3.0:1789760704.115573:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760704.115579:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760704.115591:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22065152, 22069248) append 0 append lockpos 0 00000080:00200000:3.0:1789760704.115608:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22065152=0x150b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760704.115671:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760704.115675:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22069248 00000020:00010000:3.0:1789760704.115684:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:3.0:1789760704.115687:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760704.115690:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760704.115694:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760704.115698:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760704.115719:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760704.115726:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760704.115744:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760704.115856:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af480 - object 0x240000400:5 offset 22065152<>22069248 00000080:00200000:3.0:1789760704.239737:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5387, 5387] 00000080:00200000:3.0:1789760704.239754:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760704.239761:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760704.239776:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22069248 i_blocks 45063 00000020:00200000:3.0:1789760704.239779:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760704.239788:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760704.239825:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22069248, count: 4096 00000080:00200000:3.0:1789760704.239833:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22069248, bytes: 4096 00000080:00200000:3.0:1789760704.239843:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760704.239852:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760704.239854:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22069248, 22073344) 0 0 00000080:00200000:3.0:1789760704.239858:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760704.239875:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5388, 5388] 00020000:00200000:3.0:1789760704.239884:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22069248 chunk: [22069248, 22073344) 22073344, 4096 00020000:00200000:3.0:1789760704.239888:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00010000:00010000:3.0:1789760704.285937:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760704.470180:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972708 gen 3 00000008:00010000:1.0:1789760704.470233:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760704.470242:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760704.470256:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760704.470281:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760704.470312:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512547712:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760704.471118:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512547840:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760704.538450:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512547968:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760704.577282:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972710 gen 3 00000008:00010000:1.0:1789760704.577336:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760704.577344:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760704.577357:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760704.577380:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760704.577411:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512547968:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760704.578464:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760704.578485:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512548096:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760704.660767:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760704.660783:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760704.660796:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760704.660818:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760704.660850:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512548096:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760704.662795:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512548224:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760704.825292:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972712 gen 3 00000008:00010000:1.0:1789760704.825358:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760704.825366:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760704.825378:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760704.825401:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760704.825442:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512548224:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760704.826547:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760704.826570:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512548352:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760704.965299:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972713 gen 3 00000008:00010000:1.0:1789760704.965351:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760704.965359:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760704.965383:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760704.965404:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760704.965444:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512548352:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760704.969422:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512548480:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760705.109876:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972714 gen 3 00000008:00010000:1.0:1789760705.109970:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760705.109980:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760705.109993:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760705.110016:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760705.110045:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512548480:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760705.113404:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512548608:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760705.281354:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972715 gen 3 00000008:00010000:1.0:1789760705.281415:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760705.281424:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760705.281449:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760705.281480:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760705.281512:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512548608:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760705.283970:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512548736:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760705.363978:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972716 gen 3 00000008:00010000:1.0:1789760705.364030:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760705.364039:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760705.364051:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760705.364064:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760705.364104:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512548736:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760705.365059:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc900 00000008:00100000:1.0:1789760705.365285:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 22110208<>22114304 00000100:00100000:1.0:1789760705.365341:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512548864:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760705.450990:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972717 gen 3 00000008:00010000:1.0:1789760705.451043:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760705.451052:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760705.451065:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760705.452262:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760705.452285:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512548864:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760705.635730:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512549248:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760705.703884:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972720 gen 3 00000008:00010000:1.0:1789760705.703981:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760705.703992:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760705.704005:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760705.704028:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760705.704057:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512549248:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760705.858236:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760705.863071:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512549504:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760706.119334:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5404, 5404] 00000080:00200000:1.0:1789760706.119350:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760706.119357:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760706.119374:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22138880 i_blocks 45063 00000020:00200000:1.0:1789760706.119391:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760706.119402:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760706.119439:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22138880, count: 4096 00000080:00200000:1.0:1789760706.119446:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22138880, bytes: 4096 00000080:00200000:1.0:1789760706.119456:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760706.119465:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760706.119467:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22138880, 22142976) 0 0 00000080:00200000:1.0:1789760706.119471:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760706.119489:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5405, 5405] 00020000:00200000:1.0:1789760706.119498:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22138880 chunk: [22138880, 22142976) 22142976, 4096 00020000:00200000:1.0:1789760706.119502:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760706.119522:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5405, 5405] 00000008:00010000:1.0:1789760706.119541:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760706.119547:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760706.119560:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22138880, 22142976) append 0 append lockpos 0 00000080:00200000:1.0:1789760706.119574:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22138880=0x151d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760706.119635:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760706.119638:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22142976 00000020:00010000:1.0:1789760706.119647:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:1.0:1789760706.119651:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760706.119654:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760706.119658:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760706.119660:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760706.119668:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760706.119675:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760706.119701:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:1.0:1789760706.119818:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 22138880<>22142976 00000100:00100000:1.0:1789760706.119859:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760706.119874:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512549760:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760706.236308:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972724 gen 3 00000008:00010000:1.0:1789760706.236357:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42240 00000020:00200000:1.0:1789760706.236365:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760706.236376:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760706.236390:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42240 00000100:00100000:1.0:1789760706.236771:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512549760:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760706.239529:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512549888:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760706.379797:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760706.379820:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760706.379834:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760706.379860:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760706.379891:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512549888:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760706.449134:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972726 gen 3 00000008:00010000:3.0:1789760706.449268:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760706.449277:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760706.449288:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760706.449371:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760706.449406:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512550016:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760706.450658:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512550144:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760706.518271:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972727 gen 3 00000008:00010000:3.0:1789760706.518327:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760706.518338:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760706.518353:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760706.518392:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760706.518431:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512550144:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760706.520849:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512550272:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760706.639031:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972728 gen 3 00000008:00010000:3.0:1789760706.639134:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760706.639143:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760706.639154:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760706.639172:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760706.639205:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512550272:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760706.796035:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972729 gen 3 00000008:00010000:3.0:1789760706.796093:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760706.796103:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760706.796115:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760706.796140:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760706.796170:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512550400:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760706.800294:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760706.800323:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512550528:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760706.860661:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972730 gen 3 00000008:00010000:1.0:1789760706.860724:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760706.860733:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760706.860747:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760706.860772:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760706.861719:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512550528:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760706.861886:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5411, 5411] 00000080:00200000:3.0:1789760706.861919:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760706.861924:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760706.861940:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22167552 i_blocks 45063 00000020:00200000:3.0:1789760706.861944:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760706.861955:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760706.861988:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22167552, count: 4096 00000080:00200000:3.0:1789760706.861996:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22167552, bytes: 4096 00000080:00200000:3.0:1789760706.862010:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760706.862018:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760706.862021:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22167552, 22171648) 0 0 00000080:00200000:3.0:1789760706.862025:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760706.862044:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5412, 5412] 00020000:00200000:3.0:1789760706.862053:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22167552 chunk: [22167552, 22171648) 22171648, 4096 00020000:00200000:3.0:1789760706.862057:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760706.862076:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5412, 5412] 00000008:00010000:3.0:1789760706.862093:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760706.862098:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000080:00200000:3.0:1789760706.862109:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22167552, 22171648) append 0 append lockpos 0 00000080:00200000:3.0:1789760706.863235:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22167552=0x1524000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760706.863399:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760706.863403:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22171648 00000020:00010000:3.0:1789760706.863414:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:3.0:1789760706.863417:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760706.863421:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155b30 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760706.863425:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760706.863427:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155b30 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760706.863438:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760706.863445:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760706.863469:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760706.864427:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac700 - object 0x240000400:5 offset 22167552<>22171648 00000008:00010000:3.0:1789760706.956587:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760706.956601:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760706.956615:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760706.956708:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760706.956741:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512550656:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760706.971452:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5412, 5412] 00000080:00200000:1.0:1789760706.971480:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760706.971582:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760706.971602:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22171648 i_blocks 45063 00000020:00200000:1.0:1789760706.971608:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760706.971626:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760706.971661:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22171648, count: 4096 00000080:00200000:1.0:1789760706.971669:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22171648, bytes: 4096 00000080:00200000:1.0:1789760706.971757:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760706.971768:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760706.971770:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22171648, 22175744) 0 0 00000080:00200000:1.0:1789760706.971775:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760706.971793:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5413, 5413] 00020000:00200000:1.0:1789760706.971801:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22171648 chunk: [22171648, 22175744) 22175744, 4096 00020000:00200000:1.0:1789760706.971805:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760706.971827:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5413, 5413] 00000008:00010000:1.0:1789760706.971845:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760706.971851:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760706.971862:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22171648, 22175744) append 0 append lockpos 0 00000080:00200000:1.0:1789760706.971875:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22171648=0x1525000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760706.971968:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760706.971971:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22175744 00000020:00010000:1.0:1789760706.971978:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:1.0:1789760706.971981:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760706.971984:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760706.971988:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760706.971991:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760706.971999:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760706.972005:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760706.972034:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:1.0:1789760706.975453:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 22171648<>22175744 00000100:00100000:1.0:1789760706.975562:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512550784:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760707.094787:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5413, 5413] 00000080:00200000:3.0:1789760707.094825:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760707.094831:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760707.094850:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22175744 i_blocks 45063 00000020:00200000:3.0:1789760707.094855:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760707.094874:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760707.094960:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22175744, count: 4096 00000080:00200000:3.0:1789760707.094969:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22175744, bytes: 4096 00000080:00200000:3.0:1789760707.094983:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760707.094993:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760707.094995:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22175744, 22179840) 0 0 00000080:00200000:3.0:1789760707.095000:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760707.095016:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5414, 5414] 00020000:00200000:3.0:1789760707.095024:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22175744 chunk: [22175744, 22179840) 22179840, 4096 00020000:00200000:3.0:1789760707.095028:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760707.095049:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5414, 5414] 00000008:00010000:3.0:1789760707.095067:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760707.095438:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155b30, osc lock ffff8dc010d358f0, flags 0x0 00000100:00100000:3.0:1789760707.096087:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760707.096113:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512550912:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760707.211549:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972733 gen 3 00000008:00010000:3.0:1789760707.211605:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760707.211614:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.211625:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.211646:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760707.211676:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512550912:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760707.214609:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000008:00100000:3.0:1789760707.214742:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 22179840<>22183936 00000100:00100000:3.0:1789760707.214799:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512551040:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760707.290010:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972734 gen 3 00000008:00010000:3.0:1789760707.290186:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760707.290196:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.290209:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.290290:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760707.290324:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512551040:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760707.299224:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760707.299270:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512551168:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760707.356845:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972735 gen 3 00000008:00010000:3.0:1789760707.356941:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760707.356950:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.356962:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.356976:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760707.357005:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512551168:192.168.203.156@tcp:4:dd.0 00010000:00010000:3.0:1789760707.357208:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:3.0:1789760707.422591:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760707.422606:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.422618:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.422637:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760707.422668:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512551296:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760707.512083:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5418, 5418] 00000080:00200000:3.0:1789760707.512101:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760707.512106:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760707.512122:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22196224 i_blocks 45063 00000020:00200000:3.0:1789760707.512126:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760707.512136:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760707.512171:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22196224, count: 4096 00000080:00200000:3.0:1789760707.512180:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22196224, bytes: 4096 00000080:00200000:3.0:1789760707.512191:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760707.512200:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760707.512202:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22196224, 22200320) 0 0 00000080:00200000:3.0:1789760707.512207:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760707.512224:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5419, 5419] 00020000:00200000:3.0:1789760707.512232:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22196224 chunk: [22196224, 22200320) 22200320, 4096 00020000:00200000:3.0:1789760707.512238:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760707.512258:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5419, 5419] 00000008:00010000:3.0:1789760707.512275:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760707.512280:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760707.512292:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22196224, 22200320) append 0 append lockpos 0 00000080:00200000:3.0:1789760707.512306:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22196224=0x152b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760707.512373:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760707.512376:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22200320 00000020:00010000:3.0:1789760707.512384:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:3.0:1789760707.512388:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760707.512391:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760707.512395:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760707.512398:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760707.512406:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760707.512413:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760707.512429:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760707.512532:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 22196224<>22200320 00000100:00100000:3.0:1789760707.512589:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760707.512605:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512551552:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760707.579801:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972738 gen 3 00000008:00010000:3.0:1789760707.580009:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760707.580018:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.580029:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.580053:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760707.580288:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512551552:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760707.687659:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5420, 5420] 00000080:00200000:1.0:1789760707.687684:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760707.687690:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760707.687705:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22204416 i_blocks 45063 00000020:00200000:1.0:1789760707.687709:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760707.687719:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760707.687751:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22204416, count: 4096 00000080:00200000:1.0:1789760707.687757:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22204416, bytes: 4096 00000080:00200000:1.0:1789760707.687767:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760707.687775:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760707.687778:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22204416, 22208512) 0 0 00000080:00200000:1.0:1789760707.687782:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760707.687797:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5421, 5421] 00020000:00200000:1.0:1789760707.687805:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22204416 chunk: [22204416, 22208512) 22208512, 4096 00020000:00200000:1.0:1789760707.687810:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760707.687828:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5421, 5421] 00000008:00010000:1.0:1789760707.687844:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760707.687850:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760707.687861:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22204416, 22208512) append 0 append lockpos 0 00000080:00200000:1.0:1789760707.687883:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22204416=0x152d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760707.687972:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760707.687975:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22208512 00000020:00010000:1.0:1789760707.687983:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:1.0:1789760707.687987:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760707.687990:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760707.688128:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760707.688132:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760707.688140:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760707.688146:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760707.688162:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760707.688275:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 22204416<>22208512 00000100:00100000:1.0:1789760707.688320:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512551808:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760707.777970:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972740 gen 3 00000008:00010000:1.0:1789760707.778024:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760707.778033:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760707.778044:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760707.778116:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760707.778145:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512551808:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760707.784341:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5421, 5421] 00000080:00200000:3.0:1789760707.784379:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760707.784387:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760707.784404:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22208512 i_blocks 45063 00000020:00200000:3.0:1789760707.784411:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760707.784426:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760707.784464:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22208512, count: 4096 00000080:00200000:3.0:1789760707.784473:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22208512, bytes: 4096 00000080:00200000:3.0:1789760707.784484:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760707.784494:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760707.784496:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22208512, 22212608) 0 0 00000080:00200000:3.0:1789760707.784500:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760707.784519:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5422, 5422] 00020000:00200000:3.0:1789760707.784529:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22208512 chunk: [22208512, 22212608) 22212608, 4096 00020000:00200000:3.0:1789760707.784533:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760707.784559:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5422, 5422] 00000008:00010000:3.0:1789760707.784581:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760707.784587:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760707.784601:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22208512, 22212608) append 0 append lockpos 0 00000080:00200000:3.0:1789760707.784614:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22208512=0x152e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760707.784696:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760707.784699:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22212608 00000020:00010000:3.0:1789760707.784708:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:3.0:1789760707.784712:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760707.784715:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760707.784718:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760707.784721:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760707.784730:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760707.784737:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760707.784763:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760707.784889:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 22208512<>22212608 00000100:00100000:3.0:1789760707.784984:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512551936:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760707.865979:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972741 gen 3 00000008:00010000:3.0:1789760707.866029:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760707.866104:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760707.866117:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760707.866140:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760707.866170:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512551936:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760707.866208:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5422, 5422] 00000080:00200000:1.0:1789760707.866224:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760707.866230:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760707.866245:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22212608 i_blocks 45063 00000020:00200000:1.0:1789760707.866249:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760707.866259:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760707.866292:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22212608, count: 4096 00000080:00200000:1.0:1789760707.866300:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22212608, bytes: 4096 00000080:00200000:1.0:1789760707.866310:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760707.866319:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760707.866322:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22212608, 22216704) 0 0 00000080:00200000:1.0:1789760707.866326:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760707.866345:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5423, 5423] 00020000:00200000:1.0:1789760707.866354:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22212608 chunk: [22212608, 22216704) 22216704, 4096 00020000:00200000:1.0:1789760707.866358:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760707.866378:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5423, 5423] 00000008:00010000:1.0:1789760707.866402:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760707.866408:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760707.866419:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22212608, 22216704) append 0 append lockpos 0 00000080:00200000:1.0:1789760707.866432:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22212608=0x152f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760707.866488:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760707.866491:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22216704 00000020:00010000:1.0:1789760707.866498:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:1.0:1789760707.866501:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760707.866504:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760707.866507:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760707.866510:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760707.866517:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760707.866525:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760707.866541:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760707.866657:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 22212608<>22216704 00000100:00100000:1.0:1789760707.866701:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760707.866716:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512552064:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760708.007732:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972742 gen 3 00000008:00010000:1.0:1789760708.007780:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760708.007789:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760708.007799:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760708.007823:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760708.007854:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512552064:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760708.008430:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5423, 5423] 00000080:00200000:3.0:1789760708.008449:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760708.008454:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760708.008470:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22216704 i_blocks 45063 00000020:00200000:3.0:1789760708.008475:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760708.008485:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760708.008517:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22216704, count: 4096 00000080:00200000:3.0:1789760708.008525:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22216704, bytes: 4096 00000080:00200000:3.0:1789760708.008536:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760708.008545:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760708.008548:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22216704, 22220800) 0 0 00000080:00200000:3.0:1789760708.008552:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760708.008569:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5424, 5424] 00020000:00200000:3.0:1789760708.008576:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22216704 chunk: [22216704, 22220800) 22220800, 4096 00020000:00200000:3.0:1789760708.008581:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760708.008600:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5424, 5424] 00000008:00010000:3.0:1789760708.008618:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760708.008625:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760708.008646:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22216704, 22220800) append 0 append lockpos 0 00000080:00200000:3.0:1789760708.008663:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22216704=0x1530000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760708.008733:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760708.008735:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22220800 00000020:00010000:3.0:1789760708.008744:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:3.0:1789760708.008747:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760708.008750:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760708.008753:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760708.008757:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760708.008764:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760708.008771:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760708.008788:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760708.008951:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 22216704<>22220800 00000100:00100000:3.0:1789760708.008993:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760708.009013:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512552192:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760708.054774:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972743 gen 3 00000008:00010000:3.0:1789760708.054824:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760708.054833:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760708.054844:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760708.054874:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760708.054943:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512552192:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760708.123987:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972744 gen 3 00000008:00010000:3.0:1789760708.124038:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760708.124046:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760708.124057:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760708.125826:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760708.125861:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512552320:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760708.126032:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5425, 5425] 00000080:00200000:1.0:1789760708.126048:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760708.126054:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760708.126183:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22224896 i_blocks 45063 00000020:00200000:1.0:1789760708.126287:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760708.126300:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760708.126334:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22224896, count: 4096 00000080:00200000:1.0:1789760708.126443:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22224896, bytes: 4096 00000080:00200000:1.0:1789760708.126462:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760708.126471:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760708.126473:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22224896, 22228992) 0 0 00000080:00200000:1.0:1789760708.126477:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760708.126495:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5426, 5426] 00020000:00200000:1.0:1789760708.126504:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22224896 chunk: [22224896, 22228992) 22228992, 4096 00020000:00200000:1.0:1789760708.126508:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760708.126528:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5426, 5426] 00000008:00010000:1.0:1789760708.126547:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760708.126553:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760708.126630:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22224896, 22228992) append 0 append lockpos 0 00000080:00200000:1.0:1789760708.126646:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22224896=0x1532000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760708.126723:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760708.126726:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22228992 00000020:00010000:1.0:1789760708.126736:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:1.0:1789760708.126739:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760708.126742:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760708.126745:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760708.126748:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760708.126755:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760708.126763:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760708.126781:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760708.127271:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 22224896<>22228992 00000080:00200000:3.0:1789760708.240597:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5426, 5426] 00000080:00200000:3.0:1789760708.240613:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760708.240620:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760708.240637:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22228992 i_blocks 45063 00000020:00200000:3.0:1789760708.240641:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760708.240651:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760708.240697:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22228992, count: 4096 00000080:00200000:3.0:1789760708.240704:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22228992, bytes: 4096 00000080:00200000:3.0:1789760708.240714:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760708.240724:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760708.240726:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22228992, 22233088) 0 0 00000080:00200000:3.0:1789760708.240730:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760708.240747:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5427, 5427] 00020000:00200000:3.0:1789760708.240756:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22228992 chunk: [22228992, 22233088) 22233088, 4096 00020000:00200000:3.0:1789760708.240760:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760708.240782:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5427, 5427] 00000008:00010000:3.0:1789760708.240800:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760708.240805:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760708.240817:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22228992, 22233088) append 0 append lockpos 0 00000080:00200000:3.0:1789760708.240831:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22228992=0x1533000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760708.240895:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760708.240898:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22233088 00000020:00010000:3.0:1789760708.240950:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:3.0:1789760708.240953:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760708.240956:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760708.240959:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760708.240962:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760708.240969:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760708.240976:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760708.240992:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760708.241097:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 22228992<>22233088 00000100:00100000:3.0:1789760708.241214:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760708.241231:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512552576:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760708.288407:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972746 gen 3 00000008:00010000:3.0:1789760708.288461:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760708.288470:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760708.288483:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760708.288506:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760708.288540:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512552576:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760708.327037:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972747 gen 3 00000008:00010000:3.0:1789760708.327108:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760708.327117:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760708.327128:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760708.327144:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760708.327171:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512552704:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760708.327857:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512552832:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760708.370626:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972748 gen 3 00000008:00010000:3.0:1789760708.370694:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760708.370704:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760708.370714:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760708.370781:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760708.370816:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512552832:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760708.371822:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760708.371847:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512552960:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760708.381222:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512553216:192.168.203.156@tcp:400:kworker.0 00000100:00100000:1.0:1789760708.381374:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760708.381381:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2359:1876699512553344:192.168.203.156@tcp:400:kworker.0 00000100:00100000:1.0:1789760708.395340:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2359:1876699512553344:192.168.203.156@tcp:400:kworker.0 00000100:00100000:1.0:1789760708.395430:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512553216:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760708.465202:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972749 gen 3 00000008:00010000:3.0:1789760708.465259:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760708.465268:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760708.465281:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760708.465300:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760708.465332:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512552960:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760708.470538:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512553472:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760708.589452:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760708.589469:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760708.589481:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760708.589508:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760708.589544:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512553472:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760708.723666:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512553728:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760708.787373:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972752 gen 3 00000008:00010000:3.0:1789760708.787443:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760708.787452:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760708.787461:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760708.787484:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760708.787515:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512553728:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760708.871973:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972753 gen 3 00000008:00010000:3.0:1789760708.872026:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760708.872036:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760708.872047:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760708.872076:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760708.872115:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512553856:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760708.873708:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512553984:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760708.931457:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760708.931474:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760708.931488:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760708.931510:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760708.931542:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512553984:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760708.931786:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5435, 5435] 00000080:00200000:3.0:1789760708.931804:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760708.931811:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760708.931827:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22265856 i_blocks 45063 00000020:00200000:3.0:1789760708.931831:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760708.931842:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760708.931881:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22265856, count: 4096 00000080:00200000:3.0:1789760708.931889:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22265856, bytes: 4096 00000080:00200000:3.0:1789760708.931931:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760708.931941:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760708.931944:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22265856, 22269952) 0 0 00000080:00200000:3.0:1789760708.931949:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760708.931966:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5436, 5436] 00020000:00200000:3.0:1789760708.931976:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22265856 chunk: [22265856, 22269952) 22269952, 4096 00020000:00200000:3.0:1789760708.931980:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760708.932001:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5436, 5436] 00000008:00010000:3.0:1789760708.932019:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760708.932026:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760708.932038:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22265856, 22269952) append 0 append lockpos 0 00000080:00200000:3.0:1789760708.932052:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22265856=0x153c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760708.932127:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760708.932130:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22269952 00000020:00010000:3.0:1789760708.932138:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:3.0:1789760708.932142:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760708.932145:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760708.932150:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760708.932153:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760708.932161:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760708.932168:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760708.932197:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760708.932489:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 22265856<>22269952 00000100:00100000:3.0:1789760708.932541:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760708.932982:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512554112:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760708.990452:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972755 gen 3 00000008:00010000:3.0:1789760708.990523:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760708.990532:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760708.990543:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760708.990563:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760708.990602:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512554112:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760709.081940:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972756 gen 3 00000008:00010000:3.0:1789760709.081987:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760709.081996:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760709.082007:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760709.082026:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760709.082068:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512554240:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760709.083562:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512554368:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760709.175988:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972757 gen 3 00000008:00010000:3.0:1789760709.176052:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760709.176415:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760709.176428:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760709.176447:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760709.176485:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512554368:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760709.178446:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760709.178475:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512554496:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760709.265297:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972758 gen 3 00000008:00010000:3.0:1789760709.265423:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760709.265433:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760709.265445:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760709.265542:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760709.265578:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512554496:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760709.268503:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512554624:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760709.384043:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972759 gen 3 00000008:00010000:3.0:1789760709.384095:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760709.384103:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760709.384113:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760709.384136:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760709.384167:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512554624:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760709.501584:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760709.501599:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760709.501611:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760709.501635:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760709.501684:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512554752:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760709.502694:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512554880:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760709.582293:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972761 gen 3 00000080:00200000:3.0:1789760709.591805:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5442, 5442] 00000080:00200000:3.0:1789760709.591824:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760709.591830:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760709.591845:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22294528 i_blocks 45063 00000020:00200000:3.0:1789760709.591849:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760709.591859:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760709.591891:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22294528, count: 4096 00000080:00200000:3.0:1789760709.591928:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22294528, bytes: 4096 00000080:00200000:3.0:1789760709.591944:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760709.591953:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760709.591955:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22294528, 22298624) 0 0 00000080:00200000:3.0:1789760709.591959:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760709.591976:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5443, 5443] 00020000:00200000:3.0:1789760709.591984:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22294528 chunk: [22294528, 22298624) 22298624, 4096 00020000:00200000:3.0:1789760709.591988:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760709.592008:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5443, 5443] 00000008:00010000:3.0:1789760709.592025:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760709.592031:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760709.592043:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22294528, 22298624) append 0 append lockpos 0 00000080:00200000:3.0:1789760709.592066:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22294528=0x1543000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760709.592121:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760709.592124:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22298624 00000020:00010000:3.0:1789760709.592131:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_release():245 00000020:00010000:3.0:1789760709.592135:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760709.592137:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c7e8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760709.592140:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc002d98e48 (ffff8dc018e42518/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760709.592143:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42518/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760709.592150:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760709.592156:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1789760709.654587:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972762 gen 3 00000008:00010000:3.0:1789760709.654641:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760709.654649:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760709.654669:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760709.654690:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760709.654828:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512555008:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760709.655976:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760709.655997:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512555136:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760709.739968:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972763 gen 3 00000008:00010000:3.0:1789760709.740206:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760709.740216:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760709.740226:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760709.740245:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760709.740275:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512555136:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760709.741630:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760709.741652:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512555264:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760709.797002:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972764 gen 3 00000008:00010000:3.0:1789760709.797061:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760709.797070:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760709.797081:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760709.797104:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760709.797137:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512555264:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760709.798515:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760709.798539:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512555392:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760709.886600:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972765 gen 3 00000008:00010000:3.0:1789760709.886649:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760709.886666:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760709.886679:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760709.886699:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760709.886730:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512555392:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760709.887658:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760709.887688:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512555520:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760710.028582:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972766 gen 3 00000008:00010000:3.0:1789760710.028634:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760710.028643:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.028655:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.028684:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760710.028724:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512555520:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760710.029832:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760710.029857:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512555648:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760710.136770:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972767 gen 3 00000008:00010000:3.0:1789760710.136826:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760710.136835:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.136846:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.136870:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760710.136950:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512555648:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760710.138299:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760710.138324:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512555776:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760710.194189:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972768 gen 3 00000080:00200000:3.0:1789760710.194626:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5449, 5449] 00000080:00200000:3.0:1789760710.194642:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760710.194648:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760710.194675:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22323200 i_blocks 45063 00000020:00200000:3.0:1789760710.194678:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760710.194689:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000100:00100000:3.0:1789760710.199420:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000008:00010000:3.0:1789760710.258782:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760710.258798:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.258810:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.258831:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760710.258863:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512555904:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760710.317943:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972770 gen 3 00000008:00010000:3.0:1789760710.318007:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760710.318015:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.318026:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.318048:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760710.318077:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512556032:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760710.394644:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972771 gen 3 00000008:00010000:3.0:1789760710.394702:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760710.394711:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.394788:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.394847:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760710.394879:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512556160:192.168.203.156@tcp:4:dd.0 00010000:00010000:3.0:1789760710.429618:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760710.521809:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972772 gen 3 00000008:00010000:3.0:1789760710.521863:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760710.521873:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.521883:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.521942:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760710.521984:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512556288:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760710.523225:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512556416:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760710.580222:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760710.580248:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.580260:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.580277:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760710.580308:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512556416:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760710.581532:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512556544:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760710.618433:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972774 gen 3 00000008:00010000:3.0:1789760710.618565:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760710.618574:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.618585:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.618605:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760710.618636:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512556544:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760710.619861:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512556672:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760710.686461:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972775 gen 3 00000008:00010000:3.0:1789760710.686512:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760710.686521:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.686533:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.686555:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760710.686585:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512556672:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760710.687244:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512556800:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760710.733703:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760710.733720:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.733733:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.733954:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760710.733988:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512556800:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760710.737716:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512556928:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760710.814430:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972777 gen 3 00000008:00010000:3.0:1789760710.814475:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760710.814482:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.814490:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.814507:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760710.814529:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512556928:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760710.815485:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512557056:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760710.910709:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972778 gen 3 00000008:00010000:3.0:1789760710.910758:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760710.910764:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.910774:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.910790:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760710.910814:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512557056:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760710.911436:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512557184:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760710.978738:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760710.978755:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760710.978767:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760710.978838:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760710.978872:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512557184:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760710.979661:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512557312:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760711.033713:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972779 gen 3 00000100:00100000:3.0:1789760711.033755:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368c700 x1876699512557312/t4294972780(4294972780) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760726 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760711.033812:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760711.033819:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760711.033828:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760711.033846:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760711.033869:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512557312:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760711.034683:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512557440:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760711.068251:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972781 gen 3 00000008:00010000:3.0:1789760711.068309:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760711.068316:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760711.068375:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760711.068430:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760711.068462:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512557440:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760711.069225:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512557568:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760711.097026:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972782 gen 3 00000008:00010000:3.0:1789760711.097129:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760711.097137:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760711.097149:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760711.097232:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760711.097270:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512557568:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760711.098087:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512557696:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760711.128480:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760711.128498:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760711.128514:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760711.128538:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760711.128568:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512557696:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760711.129541:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760711.129562:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512557824:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760711.202506:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972784 gen 3 00000008:00010000:1.0:1789760711.202575:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760711.202584:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760711.202594:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760711.202617:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760711.202647:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512557824:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760711.203493:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512557952:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760711.392448:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972785 gen 3 00000008:00010000:1.0:1789760711.392507:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760711.392516:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760711.392530:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760711.392558:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760711.392593:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512557952:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760711.394195:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760711.394220:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512558080:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760711.445065:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760711.445076:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760711.445085:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760711.445104:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760711.445125:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512558080:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760711.445673:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512558208:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760711.501733:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972786 gen 3 00000100:00100000:1.0:1789760711.501774:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005148e00 x1876699512558208/t4294972787(4294972787) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760727 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760711.501812:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760711.501817:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760711.501825:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760711.501844:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760711.501863:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512558208:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760711.502861:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760711.502875:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512558336:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760711.553112:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972788 gen 3 00000008:00010000:1.0:1789760711.553154:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760711.553169:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760711.553178:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760711.553196:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760711.553217:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512558336:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760711.553942:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512558464:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760711.646230:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972789 gen 3 00000008:00010000:1.0:1789760711.646266:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760711.646272:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760711.646281:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760711.646343:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760711.646366:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512558464:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760711.647043:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512558592:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760711.695546:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972790 gen 3 00000008:00010000:1.0:1789760711.695581:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760711.695588:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760711.695599:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760711.695675:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760711.695697:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512558592:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760711.696683:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512558720:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760711.771331:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760711.771342:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760711.771353:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760711.771821:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760711.771856:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512558720:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760711.779586:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512558848:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760711.859143:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972792 gen 3 00000008:00010000:3.0:1789760711.859217:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760711.859226:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760711.859237:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760711.859259:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760711.859292:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512558848:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760711.860350:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:3.0:1789760711.860461:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 22421504<>22425600 00000100:00100000:3.0:1789760711.860503:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760711.860522:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512558976:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760711.930737:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972793 gen 3 00000008:00010000:3.0:1789760711.930784:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760711.930792:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760711.930804:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760711.930821:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760711.930850:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512558976:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760712.049274:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972794 gen 3 00000008:00010000:3.0:1789760712.049310:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760712.049374:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.049384:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.049407:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760712.049433:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512559104:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760712.169823:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972795 gen 3 00000008:00010000:3.0:1789760712.169860:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760712.169867:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.169876:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.169924:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760712.169950:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512559232:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760712.171226:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512559360:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760712.299781:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972796 gen 3 00000008:00010000:1.0:1789760712.299816:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760712.299834:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760712.299844:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760712.299864:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760712.299889:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512559360:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760712.300810:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512559488:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760712.408319:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972797 gen 3 00000008:00010000:1.0:1789760712.408357:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760712.408365:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760712.408376:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760712.408398:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760712.408420:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512559488:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760712.409298:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760712.409313:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512559616:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760712.455144:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972798 gen 3 00000008:00010000:3.0:1789760712.455184:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760712.455191:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.455201:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.455219:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760712.455245:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512559616:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760712.455933:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512559744:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760712.513193:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972799 gen 3 00000008:00010000:3.0:1789760712.513228:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760712.513234:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.513244:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.513262:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760712.513285:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512559744:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760712.514019:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760712.514041:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512559872:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760712.566804:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972800 gen 3 00000008:00010000:3.0:1789760712.566838:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760712.566845:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.566853:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.566870:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760712.566894:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512559872:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760712.568649:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512560000:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760712.625593:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972801 gen 3 00000008:00010000:3.0:1789760712.625631:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760712.625645:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.625655:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.625670:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760712.625693:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512560000:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760712.626933:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760712.626959:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512560128:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760712.675849:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760712.675862:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.675871:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.675890:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760712.675943:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512560128:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760712.676831:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512560256:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760712.727979:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972802 gen 3 00000100:00100000:3.0:1789760712.728008:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368c380 x1876699512560256/t4294972803(4294972803) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760728 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760712.728055:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760712.728061:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.728077:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.728092:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760712.728113:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512560256:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760712.728757:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512560384:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760712.803600:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972804 gen 3 00000008:00010000:3.0:1789760712.803657:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760712.803664:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.803671:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.803686:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760712.803710:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512560384:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760712.804343:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512560512:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760712.878119:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760712.878137:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.878146:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.878165:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760712.878185:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512560512:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760712.878750:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512560640:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760712.950587:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972805 gen 3 00000100:00100000:3.0:1789760712.950618:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368c000 x1876699512560640/t4294972806(4294972806) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760728 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760712.950669:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760712.950676:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760712.950685:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760712.950704:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760712.950721:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512560640:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760712.951557:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512560768:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.056573:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972806 gen 3 00000100:00100000:3.0:1789760713.056603:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368f480 x1876699512560768/t4294972807(4294972807) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760728 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760713.056650:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760713.056656:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.056663:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.056680:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760713.056701:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512560768:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.057204:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512560896:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.154077:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972807 gen 3 00000100:00100000:3.0:1789760713.154105:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368fb80 x1876699512560896/t4294972808(4294972808) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760729 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760713.154141:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760713.154147:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.154155:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.154170:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760713.154189:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512560896:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.154756:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512561024:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.189759:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972808 gen 3 00000100:00100000:3.0:1789760713.189792:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368d880 x1876699512561024/t4294972809(4294972809) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760729 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760713.189828:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760713.189834:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.189841:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.189857:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760713.189875:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512561024:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.190394:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512561152:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.248614:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972809 gen 3 00000100:00100000:3.0:1789760713.248655:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368f100 x1876699512561152/t4294972810(4294972810) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760729 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760713.248692:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760713.248698:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.248706:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.248723:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760713.248744:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512561152:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.250184:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512561280:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.298995:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972811 gen 3 00000008:00010000:3.0:1789760713.299069:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760713.299077:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.299087:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.299105:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760713.299136:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512561280:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.300104:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512561408:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760713.373581:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760713.373597:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.373608:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.373627:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760713.373664:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512561408:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.375440:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512561536:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.429811:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972812 gen 3 00000100:00100000:3.0:1789760713.429841:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368f480 x1876699512561536/t4294972813(4294972813) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760729 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760713.429877:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760713.429882:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.429889:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.430010:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760713.430039:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512561536:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.431299:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512561664:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.472268:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972814 gen 3 00000008:00010000:3.0:1789760713.472321:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760713.472328:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.472336:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.472349:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760713.472372:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512561664:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.473118:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512561792:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.501935:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512562048:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760713.502021:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1789760713.502035:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e6cf7267-9b19-462a-a263-14115df15dab:2361:1876699512561920:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760713.502061:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2361:1876699512562176:192.168.203.156@tcp:400:kworker.0 00010000:00010000:3.0:1789760713.502857:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760713.505538:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e6cf7267-9b19-462a-a263-14115df15dab:2361:1876699512561920:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760713.505557:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2361:1876699512562176:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760713.505566:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512562048:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760713.517403:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972815 gen 3 00000008:00010000:3.0:1789760713.517441:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760713.517448:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.517456:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.517474:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760713.517498:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512561792:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.518087:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512562304:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.575769:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972816 gen 3 00000008:00010000:3.0:1789760713.575810:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760713.575816:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.575824:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.575841:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760713.575864:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512562304:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.577467:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512562432:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.624414:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972817 gen 3 00000008:00010000:3.0:1789760713.624450:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760713.624456:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.624465:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.624482:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760713.624506:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512562432:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.625131:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760713.625146:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512562560:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.684608:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972818 gen 3 00000008:00010000:3.0:1789760713.684656:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760713.684662:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.684671:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.684687:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760713.684709:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512562560:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.685319:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512562688:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.765380:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972819 gen 3 00000008:00010000:3.0:1789760713.765415:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760713.765422:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.765430:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.765452:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760713.765474:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512562688:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.766126:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760713.766140:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512562816:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.845873:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972820 gen 3 00000008:00010000:3.0:1789760713.845939:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760713.845946:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.845955:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.845973:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760713.845996:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512562816:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.847052:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760713.847064:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512562944:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.912850:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972821 gen 3 00000008:00010000:3.0:1789760713.912887:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760713.912894:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.912927:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.912947:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760713.912969:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512562944:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760713.913598:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760713.913611:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512563072:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760713.982211:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760713.982231:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760713.982240:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760713.982258:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760713.982281:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512563072:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760714.027791:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972823 gen 3 00000008:00010000:3.0:1789760714.027836:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760714.027843:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760714.027851:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760714.027874:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760714.027898:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512563200:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760714.028487:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512563328:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760714.079185:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972824 gen 3 00000008:00010000:1.0:1789760714.079223:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760714.079230:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760714.079240:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760714.079270:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760714.079294:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512563328:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760714.080446:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512563456:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760714.139994:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760714.140005:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760714.140014:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760714.140031:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760714.140055:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512563456:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760714.140843:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760714.140859:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512563584:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760714.196777:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972826 gen 3 00000008:00010000:1.0:1789760714.196826:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760714.196833:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760714.196841:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760714.196863:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760714.196885:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512563584:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760714.198392:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760714.198409:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512563712:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760714.259938:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972827 gen 3 00000008:00010000:1.0:1789760714.259975:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760714.259982:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760714.259992:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760714.260012:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760714.260036:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512563712:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760714.261137:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512563840:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760714.327556:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972828 gen 3 00000008:00010000:1.0:1789760714.327591:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760714.327597:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760714.327607:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760714.327619:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760714.328118:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512563840:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760714.331304:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512563968:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760714.385852:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972829 gen 3 00000008:00010000:1.0:1789760714.385886:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760714.385893:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760714.385936:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760714.385946:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760714.385968:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512563968:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760714.386877:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760714.386893:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512564096:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760714.448290:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972830 gen 3 00000008:00010000:1.0:1789760714.448324:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760714.448331:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760714.448340:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760714.448478:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760714.448503:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512564096:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760714.452027:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512564224:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760714.512392:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972831 gen 3 00000008:00010000:1.0:1789760714.512427:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760714.512433:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760714.512441:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760714.512462:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760714.512486:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512564224:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760714.513868:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5512, 5512] 00000080:00200000:3.0:1789760714.513885:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760714.513890:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760714.513927:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22581248 i_blocks 45063 00000020:00200000:3.0:1789760714.513932:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760714.513942:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760714.513968:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22581248, count: 4096 00000080:00200000:3.0:1789760714.513974:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22581248, bytes: 4096 00000080:00200000:3.0:1789760714.513984:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760714.513993:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760714.513995:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22581248, 22585344) 0 0 00000080:00200000:3.0:1789760714.513999:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760714.514012:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5513, 5513] 00020000:00200000:3.0:1789760714.514029:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22581248 chunk: [22581248, 22585344) 22585344, 4096 00020000:00200000:3.0:1789760714.514033:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760714.514054:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5513, 5513] 00000008:00010000:3.0:1789760714.514069:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760714.514074:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760714.514083:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22581248, 22585344) append 0 append lockpos 0 00000080:00200000:3.0:1789760714.514094:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22581248=0x1589000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760714.514142:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760714.514145:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22585344 00000020:00010000:3.0:1789760714.514151:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760714.514155:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760714.514158:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760714.514161:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760714.514163:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760714.514170:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760714.514176:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760714.514192:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760714.514273:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 22581248<>22585344 00000100:00100000:3.0:1789760714.514322:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760714.514338:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512564352:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760714.594581:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760714.594593:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760714.594603:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760714.594633:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760714.594665:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512564352:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760714.595122:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5513, 5513] 00000080:00200000:1.0:1789760714.595134:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760714.595139:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760714.595149:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22585344 i_blocks 45063 00000020:00200000:1.0:1789760714.595152:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760714.595159:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760714.595194:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22585344, count: 4096 00000080:00200000:1.0:1789760714.595200:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22585344, bytes: 4096 00000080:00200000:1.0:1789760714.595208:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760714.595215:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760714.595217:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22585344, 22589440) 0 0 00000080:00200000:1.0:1789760714.595221:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760714.595232:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5514, 5514] 00020000:00200000:1.0:1789760714.595238:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22585344 chunk: [22585344, 22589440) 22589440, 4096 00020000:00200000:1.0:1789760714.595241:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760714.595255:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5514, 5514] 00000008:00010000:1.0:1789760714.595267:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760714.595271:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760714.595279:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22585344, 22589440) append 0 append lockpos 0 00000080:00200000:1.0:1789760714.595289:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22585344=0x158a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760714.595331:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760714.595334:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22589440 00000020:00010000:1.0:1789760714.595340:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760714.595344:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760714.595346:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760714.595350:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760714.595352:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760714.595369:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760714.595374:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760714.595387:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760714.595466:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 22585344<>22589440 00000100:00100000:1.0:1789760714.595505:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760714.595520:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512564480:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760714.673576:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972833 gen 3 00000008:00010000:1.0:1789760714.673620:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760714.673637:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760714.673646:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760714.673711:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760714.673733:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512564480:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760714.677760:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5514, 5514] 00000080:00200000:3.0:1789760714.677780:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760714.677785:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760714.677811:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22589440 i_blocks 45063 00000020:00200000:3.0:1789760714.677815:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760714.677829:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760714.677856:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22589440, count: 4096 00000080:00200000:3.0:1789760714.677863:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22589440, bytes: 4096 00000080:00200000:3.0:1789760714.677875:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760714.677881:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760714.677883:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22589440, 22593536) 0 0 00000080:00200000:3.0:1789760714.677887:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760714.677916:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5515, 5515] 00020000:00200000:3.0:1789760714.677924:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22589440 chunk: [22589440, 22593536) 22593536, 4096 00020000:00200000:3.0:1789760714.677928:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760714.677945:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5515, 5515] 00000008:00010000:3.0:1789760714.677960:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760714.677965:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760714.677975:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22589440, 22593536) append 0 append lockpos 0 00000080:00200000:3.0:1789760714.677987:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22589440=0x158b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760714.678051:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760714.678055:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22593536 00000020:00010000:3.0:1789760714.678062:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760714.678065:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760714.678067:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760714.678070:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760714.678074:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760714.678081:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760714.678087:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760714.678109:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760714.678401:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107afb80 - object 0x240000400:5 offset 22589440<>22593536 00000100:00100000:3.0:1789760714.728274:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972834 gen 3 00000008:00010000:3.0:1789760714.728318:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760714.728325:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760714.728335:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760714.728356:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760714.728380:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512564608:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760714.729330:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5515, 5515] 00000080:00200000:1.0:1789760714.729346:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760714.729351:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760714.729362:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22593536 i_blocks 45063 00000020:00200000:1.0:1789760714.729366:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760714.729375:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760714.729400:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22593536, count: 4096 00000080:00200000:1.0:1789760714.729406:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22593536, bytes: 4096 00000080:00200000:1.0:1789760714.729416:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760714.729423:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760714.729425:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22593536, 22597632) 0 0 00000080:00200000:1.0:1789760714.729429:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760714.729443:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5516, 5516] 00020000:00200000:1.0:1789760714.729450:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22593536 chunk: [22593536, 22597632) 22597632, 4096 00020000:00200000:1.0:1789760714.729453:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760714.729469:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5516, 5516] 00000008:00010000:1.0:1789760714.729483:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760714.729488:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760714.729495:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22593536, 22597632) append 0 append lockpos 0 00000080:00200000:1.0:1789760714.729506:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22593536=0x158c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760714.729556:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760714.729558:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22597632 00000020:00010000:1.0:1789760714.729565:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760714.729568:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760714.729571:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760714.729573:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760714.729576:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760714.729582:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760714.729587:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760714.729608:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760714.729696:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 22593536<>22597632 00000080:00200000:1.0:1789760714.783635:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5516, 5516] 00000080:00200000:1.0:1789760714.783648:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760714.783653:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760714.783664:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22597632 i_blocks 45063 00000020:00200000:1.0:1789760714.783668:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760714.783676:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760714.783702:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22597632, count: 4096 00000080:00200000:1.0:1789760714.783708:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22597632, bytes: 4096 00000080:00200000:1.0:1789760714.783716:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760714.783723:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760714.783725:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22597632, 22601728) 0 0 00000080:00200000:1.0:1789760714.783729:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760714.783743:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5517, 5517] 00020000:00200000:1.0:1789760714.783750:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22597632 chunk: [22597632, 22601728) 22601728, 4096 00020000:00200000:1.0:1789760714.783753:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760714.783767:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5517, 5517] 00000008:00010000:1.0:1789760714.783780:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760714.783785:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760714.783794:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22597632, 22601728) append 0 append lockpos 0 00000080:00200000:1.0:1789760714.783806:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22597632=0x158d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760714.783851:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760714.783854:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22601728 00000020:00010000:1.0:1789760714.783861:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760714.783864:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760714.783867:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760714.783870:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760714.783873:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760714.783879:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760714.783885:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760714.783933:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:1.0:1789760714.784026:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 22597632<>22601728 00000100:00100000:1.0:1789760714.784059:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760714.784072:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512564864:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760714.856482:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972836 gen 3 00000008:00010000:1.0:1789760714.856542:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42240 00000020:00200000:1.0:1789760714.856548:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760714.856556:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760714.856577:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42240 00000100:00100000:1.0:1789760714.856596:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512564864:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760714.856837:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5517, 5517] 00000080:00200000:3.0:1789760714.856849:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760714.856855:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760714.856866:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22601728 i_blocks 45063 00000020:00200000:3.0:1789760714.856869:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760714.856877:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760714.856927:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22601728, count: 4096 00000080:00200000:3.0:1789760714.856934:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22601728, bytes: 4096 00000080:00200000:3.0:1789760714.856942:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760714.856949:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760714.856951:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22601728, 22605824) 0 0 00000080:00200000:3.0:1789760714.856956:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760714.856968:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5518, 5518] 00020000:00200000:3.0:1789760714.856975:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22601728 chunk: [22601728, 22605824) 22605824, 4096 00020000:00200000:3.0:1789760714.856978:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760714.856993:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5518, 5518] 00000008:00010000:3.0:1789760714.857006:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760714.857011:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760714.857021:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22601728, 22605824) append 0 append lockpos 0 00000080:00200000:3.0:1789760714.857032:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22601728=0x158e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760714.857075:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760714.857078:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22605824 00000020:00010000:3.0:1789760714.857084:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760714.857087:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760714.857090:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760714.857093:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760714.857095:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760714.857102:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760714.857107:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760714.857120:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760714.857194:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107afb80 - object 0x240000400:5 offset 22601728<>22605824 00000100:00100000:3.0:1789760714.935278:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972837 gen 3 00000008:00010000:3.0:1789760714.935324:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760714.935331:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760714.935341:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760714.935360:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760714.935381:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512564992:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760714.992499:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5519, 5519] 00000080:00200000:1.0:1789760714.992512:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760714.992516:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760714.992528:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22609920 i_blocks 45063 00000020:00200000:1.0:1789760714.992531:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760714.992539:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760714.992566:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22609920, count: 4096 00000080:00200000:1.0:1789760714.992572:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22609920, bytes: 4096 00000080:00200000:1.0:1789760714.992580:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760714.992587:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760714.992589:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22609920, 22614016) 0 0 00000080:00200000:1.0:1789760714.992593:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760714.992605:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5520, 5520] 00020000:00200000:1.0:1789760714.992613:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22609920 chunk: [22609920, 22614016) 22614016, 4096 00020000:00200000:1.0:1789760714.992616:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760714.992646:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5520, 5520] 00000008:00010000:1.0:1789760714.992660:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760714.992664:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760714.992674:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22609920, 22614016) append 0 append lockpos 0 00000080:00200000:1.0:1789760714.992685:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22609920=0x1590000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760714.992727:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760714.992730:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22614016 00000020:00010000:1.0:1789760714.992736:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760714.992739:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760714.992742:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760714.992744:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760714.992747:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760714.992753:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760714.992758:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760714.992771:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:1.0:1789760714.992846:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 22609920<>22614016 00000100:00100000:1.0:1789760714.992883:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760714.992896:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512565248:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760715.055678:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972839 gen 3 00000008:00010000:1.0:1789760715.055719:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42240 00000020:00200000:1.0:1789760715.055726:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760715.055735:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760715.055758:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42240 00000100:00100000:1.0:1789760715.055783:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512565248:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760715.124407:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5521, 5521] 00000080:00200000:1.0:1789760715.124418:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760715.124424:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760715.124436:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22618112 i_blocks 45063 00000020:00200000:1.0:1789760715.124439:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760715.124447:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760715.124473:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22618112, count: 4096 00000080:00200000:1.0:1789760715.124478:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22618112, bytes: 4096 00000080:00200000:1.0:1789760715.124485:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760715.124492:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760715.124494:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22618112, 22622208) 0 0 00000080:00200000:1.0:1789760715.124497:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760715.124509:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5522, 5522] 00020000:00200000:1.0:1789760715.124515:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22618112 chunk: [22618112, 22622208) 22622208, 4096 00020000:00200000:1.0:1789760715.124519:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760715.124532:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5522, 5522] 00000008:00010000:1.0:1789760715.124543:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760715.124548:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760715.124556:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22618112, 22622208) append 0 append lockpos 0 00000080:00200000:1.0:1789760715.124566:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22618112=0x1592000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760715.124604:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760715.124606:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22622208 00000020:00010000:1.0:1789760715.124612:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760715.124615:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760715.124618:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760715.124639:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760715.124641:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760715.124647:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760715.124653:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760715.124666:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:1.0:1789760715.124740:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 22618112<>22622208 00000100:00100000:1.0:1789760715.124774:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760715.124786:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512565504:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760715.173361:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972841 gen 3 00000008:00010000:1.0:1789760715.173399:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42240 00000020:00200000:1.0:1789760715.173405:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760715.173413:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760715.173434:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42240 00000100:00100000:1.0:1789760715.173459:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512565504:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760715.174398:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512565632:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760715.249038:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760715.249059:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760715.249074:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760715.249100:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760715.249136:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512565632:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760715.251096:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512565760:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760715.330175:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972843 gen 3 00000008:00010000:1.0:1789760715.330247:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760715.330256:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760715.330268:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760715.330328:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760715.330369:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512565760:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760715.331141:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512565888:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760715.395437:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760715.395451:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760715.395463:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760715.395488:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760715.395522:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512565888:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760715.396397:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512566016:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760715.452238:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972845 gen 3 00000080:00200000:1.0:1789760715.452662:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5526, 5526] 00000080:00200000:1.0:1789760715.452676:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760715.452681:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760715.452693:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22638592 i_blocks 45063 00000020:00200000:1.0:1789760715.452696:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760715.452704:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760715.452729:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22638592, count: 4096 00000080:00200000:1.0:1789760715.452736:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22638592, bytes: 4096 00000080:00200000:1.0:1789760715.452746:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760715.452753:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760715.452755:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22638592, 22642688) 0 0 00000080:00200000:1.0:1789760715.452758:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760715.452768:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5527, 5527] 00020000:00200000:1.0:1789760715.452773:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22638592 chunk: [22638592, 22642688) 22642688, 4096 00020000:00200000:1.0:1789760715.452776:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760715.452788:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5527, 5527] 00000008:00010000:1.0:1789760715.452800:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760715.452803:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760715.452810:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22638592, 22642688) append 0 append lockpos 0 00000080:00200000:1.0:1789760715.452819:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22638592=0x1597000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760715.452850:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760715.452852:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22642688 00000020:00010000:1.0:1789760715.452856:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760715.452858:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760715.452860:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760715.452862:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760715.452864:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760715.452868:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760715.452872:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1789760715.453522:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512566144:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760715.523165:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972846 gen 3 00000008:00010000:3.0:1789760715.523204:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42240 00000020:00200000:3.0:1789760715.523210:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760715.523219:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760715.523241:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42240 00000100:00100000:3.0:1789760715.523266:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512566144:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760715.523841:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5527, 5527] 00000080:00200000:1.0:1789760715.523854:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760715.523859:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760715.523870:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22642688 i_blocks 45063 00000020:00200000:1.0:1789760715.523872:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760715.523880:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760715.523939:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22642688, count: 4096 00000080:00200000:1.0:1789760715.523946:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22642688, bytes: 4096 00000080:00200000:1.0:1789760715.523954:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760715.523961:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760715.523963:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22642688, 22646784) 0 0 00000080:00200000:1.0:1789760715.523966:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760715.523978:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5528, 5528] 00020000:00200000:1.0:1789760715.523984:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22642688 chunk: [22642688, 22646784) 22646784, 4096 00020000:00200000:1.0:1789760715.523987:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760715.524002:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5528, 5528] 00000008:00010000:1.0:1789760715.524014:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760715.524018:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760715.524027:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22642688, 22646784) append 0 append lockpos 0 00000080:00200000:1.0:1789760715.524038:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22642688=0x1598000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760715.524079:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760715.524082:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22646784 00000020:00010000:1.0:1789760715.524088:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760715.524091:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760715.524094:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760715.524097:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760715.524100:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760715.524106:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760715.524110:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760715.524142:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42900 00000008:00100000:1.0:1789760715.524217:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 22642688<>22646784 00000100:00100000:1.0:1789760715.524257:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760715.524272:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512566272:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760715.602578:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972847 gen 3 00000008:00010000:1.0:1789760715.602626:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42900 00000020:00200000:1.0:1789760715.602631:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760715.602640:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760715.602660:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42900 00000100:00100000:1.0:1789760715.602682:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512566272:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760715.604043:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512566400:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760715.670952:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972848 gen 3 00000008:00010000:3.0:1789760715.670993:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760715.671000:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760715.671018:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760715.671033:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760715.671057:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512566400:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760715.673112:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760715.673142:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512566528:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760715.752957:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972849 gen 3 00000008:00010000:3.0:1789760715.752997:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760715.753003:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760715.753494:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760715.753515:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760715.753542:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512566528:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760715.756585:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760715.756607:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512566656:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760715.838134:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972850 gen 3 00000008:00010000:3.0:1789760715.838171:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760715.838177:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760715.838185:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760715.838203:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760715.838227:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512566656:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760715.915824:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972851 gen 3 00000008:00010000:3.0:1789760715.915859:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760715.915866:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760715.915874:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760715.915893:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760715.915943:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512566784:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760715.916852:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512566912:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.019949:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972852 gen 3 00000008:00010000:3.0:1789760716.019985:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760716.019992:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.020001:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.020026:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760716.020052:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512566912:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.021130:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512567040:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760716.069030:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760716.069042:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.069052:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.069069:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760716.069092:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512567040:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.070420:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512567168:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.115943:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972854 gen 3 00000008:00010000:3.0:1789760716.115987:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760716.115994:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.116002:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.116107:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760716.116141:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512567168:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.116892:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512567296:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.167955:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972855 gen 3 00000008:00010000:3.0:1789760716.167990:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760716.167997:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.168017:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.168033:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760716.168056:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512567296:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.168741:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512567424:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.225520:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972856 gen 3 00000008:00010000:3.0:1789760716.225556:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760716.225562:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.225571:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.225588:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760716.225610:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512567424:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.226300:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512567552:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.279018:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972857 gen 3 00000008:00010000:3.0:1789760716.279053:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760716.279059:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.279067:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.279083:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760716.279103:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512567552:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760716.279606:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:3.0:1789760716.279690:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107afb80 - object 0x240000400:5 offset 22687744<>22691840 00000100:00100000:3.0:1789760716.279745:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512567680:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.316192:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972858 gen 3 00000008:00010000:3.0:1789760716.316242:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760716.316250:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.316261:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.316281:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760716.316313:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512567680:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.318643:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512567808:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.417477:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972859 gen 3 00000008:00010000:3.0:1789760716.417528:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760716.417537:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.417548:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.417570:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760716.417601:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512567808:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.418591:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512567936:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.475351:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972860 gen 3 00000008:00010000:3.0:1789760716.475405:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760716.475413:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.475424:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.475444:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760716.475475:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512567936:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.477497:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512568064:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.547309:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972861 gen 3 00000008:00010000:3.0:1789760716.547454:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760716.547460:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.547469:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.547533:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760716.547558:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512568064:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.548302:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512568192:192.168.203.156@tcp:4:dd.0 00010000:00010000:3.0:1789760716.578290:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760716.587791:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972862 gen 3 00000008:00010000:3.0:1789760716.587825:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760716.587831:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.587838:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.587857:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760716.587880:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512568192:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760716.588435:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000008:00100000:3.0:1789760716.588510:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107afb80 - object 0x240000400:5 offset 22708224<>22712320 00000100:00100000:3.0:1789760716.588546:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760716.588558:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512568320:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760716.639638:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760716.639649:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.639658:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.639676:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760716.640541:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512568320:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760716.640567:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:3.0:1789760716.640658:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 22712320<>22716416 00000100:00100000:3.0:1789760716.640767:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512568448:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.672969:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972864 gen 3 00000008:00010000:3.0:1789760716.673027:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760716.673034:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.673042:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.673063:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760716.673088:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512568448:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.674086:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512568576:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.727547:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972865 gen 3 00000008:00010000:3.0:1789760716.727581:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760716.727587:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.727596:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.727622:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760716.727646:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512568576:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.728765:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760716.728783:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512568704:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.797173:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972866 gen 3 00000008:00010000:3.0:1789760716.797229:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760716.797236:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.797245:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.797262:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760716.797289:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512568704:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.798409:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512568832:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.873021:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972867 gen 3 00000008:00010000:3.0:1789760716.873060:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760716.873067:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.873075:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.873093:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760716.873118:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512568832:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.873988:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512568960:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.924263:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972868 gen 3 00000008:00010000:3.0:1789760716.924311:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760716.924318:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.924326:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.924342:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760716.924365:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512568960:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.925140:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512569088:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.988188:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972869 gen 3 00000008:00010000:3.0:1789760716.988225:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760716.988231:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760716.988240:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760716.988258:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760716.988287:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512569088:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760716.989204:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512569216:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760717.097710:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972870 gen 3 00000008:00010000:3.0:1789760717.097805:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760717.097812:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760717.097821:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760717.097844:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760717.097868:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512569216:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760717.097935:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5551, 5551] 00000080:00200000:1.0:1789760717.097948:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760717.097953:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760717.097966:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22740992 i_blocks 45063 00000020:00200000:1.0:1789760717.097970:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760717.097978:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760717.098001:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22740992, count: 4096 00000080:00200000:1.0:1789760717.098007:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22740992, bytes: 4096 00000080:00200000:1.0:1789760717.098015:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760717.098022:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760717.098024:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22740992, 22745088) 0 0 00000080:00200000:1.0:1789760717.098028:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760717.098041:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5552, 5552] 00020000:00200000:1.0:1789760717.098047:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22740992 chunk: [22740992, 22745088) 22745088, 4096 00020000:00200000:1.0:1789760717.098051:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760717.098108:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5552, 5552] 00000008:00010000:1.0:1789760717.098124:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760717.098129:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760717.098138:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22740992, 22745088) append 0 append lockpos 0 00000080:00200000:1.0:1789760717.098150:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22740992=0x15b0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760717.098193:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760717.098196:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22745088 00000020:00010000:1.0:1789760717.098202:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760717.098205:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760717.098207:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760717.098210:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760717.098212:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760717.098218:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760717.098224:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760717.098237:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42900 00000008:00100000:1.0:1789760717.098315:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 22740992<>22745088 00000080:00200000:1.0:1789760717.170289:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5552, 5552] 00000080:00200000:1.0:1789760717.170301:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760717.170306:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760717.170317:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22745088 i_blocks 45063 00000020:00200000:1.0:1789760717.170321:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760717.170328:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760717.170354:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22745088, count: 4096 00000080:00200000:1.0:1789760717.170360:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22745088, bytes: 4096 00000080:00200000:1.0:1789760717.170368:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760717.170374:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760717.170376:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22745088, 22749184) 0 0 00000080:00200000:1.0:1789760717.170381:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760717.170393:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5553, 5553] 00020000:00200000:1.0:1789760717.170399:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22745088 chunk: [22745088, 22749184) 22749184, 4096 00020000:00200000:1.0:1789760717.170403:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760717.170418:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5553, 5553] 00000008:00010000:1.0:1789760717.170430:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760717.170435:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760717.170443:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22745088, 22749184) append 0 append lockpos 0 00000080:00200000:1.0:1789760717.170453:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22745088=0x15b1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760717.170494:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760717.170497:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22749184 00000020:00010000:1.0:1789760717.170503:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760717.170506:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760717.170508:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760717.170511:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760717.170514:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760717.170520:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760717.170526:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760717.170538:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f429c0 00000008:00100000:1.0:1789760717.171747:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 22745088<>22749184 00000100:00100000:1.0:1789760717.171794:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760717.171810:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512569472:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760717.244484:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972872 gen 3 00000008:00010000:1.0:1789760717.244530:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f429c0 00000020:00200000:1.0:1789760717.244536:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760717.244544:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760717.244564:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f429c0 00000100:00100000:1.0:1789760717.244595:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512569472:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760717.333190:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5554, 5554] 00000080:00200000:1.0:1789760717.333203:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760717.333208:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760717.333219:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22753280 i_blocks 45063 00000020:00200000:1.0:1789760717.333223:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760717.333231:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760717.333260:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22753280, count: 4096 00000080:00200000:1.0:1789760717.333266:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22753280, bytes: 4096 00000080:00200000:1.0:1789760717.333274:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760717.333281:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760717.333283:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22753280, 22757376) 0 0 00000080:00200000:1.0:1789760717.333288:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760717.333301:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5555, 5555] 00020000:00200000:1.0:1789760717.333308:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22753280 chunk: [22753280, 22757376) 22757376, 4096 00020000:00200000:1.0:1789760717.333312:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760717.333327:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5555, 5555] 00000008:00010000:1.0:1789760717.333341:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760717.333346:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760717.333356:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22753280, 22757376) append 0 append lockpos 0 00000080:00200000:1.0:1789760717.333367:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22753280=0x15b3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760717.333412:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760717.333414:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22757376 00000020:00010000:1.0:1789760717.333421:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760717.333424:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760717.333427:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760717.333430:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760717.333432:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760717.333438:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760717.333444:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760717.333458:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f429c0 00000008:00100000:1.0:1789760717.333538:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 22753280<>22757376 00000100:00100000:1.0:1789760717.333579:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760717.333593:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512569728:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760717.378074:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972874 gen 3 00000008:00010000:1.0:1789760717.378109:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f429c0 00000020:00200000:1.0:1789760717.378115:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760717.378123:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760717.378149:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f429c0 00000100:00100000:1.0:1789760717.378170:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512569728:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760717.378292:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5555, 5555] 00000080:00200000:3.0:1789760717.378306:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760717.378312:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760717.378327:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22757376 i_blocks 45063 00000020:00200000:3.0:1789760717.378330:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760717.378339:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760717.378363:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22757376, count: 4096 00000080:00200000:3.0:1789760717.378380:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22757376, bytes: 4096 00000080:00200000:3.0:1789760717.378389:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760717.378397:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760717.378399:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22757376, 22761472) 0 0 00000080:00200000:3.0:1789760717.378402:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760717.378415:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5556, 5556] 00020000:00200000:3.0:1789760717.378422:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22757376 chunk: [22757376, 22761472) 22761472, 4096 00020000:00200000:3.0:1789760717.378436:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760717.378453:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5556, 5556] 00000008:00010000:3.0:1789760717.378467:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760717.378472:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760717.378482:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22757376, 22761472) append 0 append lockpos 0 00000080:00200000:3.0:1789760717.378492:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22757376=0x15b4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760717.378542:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760717.378544:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22761472 00000020:00010000:3.0:1789760717.378551:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760717.378554:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760717.378557:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760717.378559:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760717.378562:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760717.378567:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760717.378573:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760717.378586:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760717.378674:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 22757376<>22761472 00000100:00100000:3.0:1789760717.378715:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760717.378728:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512569856:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760717.428592:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972875 gen 3 00000008:00010000:3.0:1789760717.428636:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760717.428642:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760717.428650:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760717.428670:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760717.428691:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512569856:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760717.428789:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5556, 5556] 00000080:00200000:1.0:1789760717.428802:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760717.428807:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760717.428819:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22761472 i_blocks 45063 00000020:00200000:1.0:1789760717.428822:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760717.428830:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760717.428858:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22761472, count: 4096 00000080:00200000:1.0:1789760717.428864:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22761472, bytes: 4096 00000080:00200000:1.0:1789760717.428873:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760717.428881:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760717.428883:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22761472, 22765568) 0 0 00000080:00200000:1.0:1789760717.428887:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760717.429099:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5557, 5557] 00020000:00200000:1.0:1789760717.429108:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22761472 chunk: [22761472, 22765568) 22765568, 4096 00020000:00200000:1.0:1789760717.429111:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760717.429130:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5557, 5557] 00000008:00010000:1.0:1789760717.429143:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760717.429148:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760717.429157:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22761472, 22765568) append 0 append lockpos 0 00000080:00200000:1.0:1789760717.429167:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22761472=0x15b5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760717.429212:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760717.429214:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22765568 00000020:00010000:1.0:1789760717.429221:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760717.429225:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760717.429228:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760717.429230:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760717.429233:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760717.429239:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760717.429244:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760717.429259:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f429c0 00000008:00100000:1.0:1789760717.429345:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 22761472<>22765568 00000080:00200000:1.0:1789760717.501405:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5557, 5557] 00000080:00200000:1.0:1789760717.501418:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760717.501423:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760717.501442:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22765568 i_blocks 45063 00000020:00200000:1.0:1789760717.501446:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760717.501453:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760717.501480:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22765568, count: 4096 00000080:00200000:1.0:1789760717.501486:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22765568, bytes: 4096 00000080:00200000:1.0:1789760717.501494:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760717.501501:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760717.501503:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22765568, 22769664) 0 0 00000080:00200000:1.0:1789760717.501507:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760717.501521:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5558, 5558] 00020000:00200000:1.0:1789760717.501528:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22765568 chunk: [22765568, 22769664) 22769664, 4096 00020000:00200000:1.0:1789760717.501532:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760717.501549:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5558, 5558] 00000008:00010000:1.0:1789760717.501565:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760717.501570:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760717.501580:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22765568, 22769664) append 0 append lockpos 0 00000080:00200000:1.0:1789760717.501590:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22765568=0x15b6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760717.501646:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760717.501648:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22769664 00000020:00010000:1.0:1789760717.501656:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760717.501659:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760717.501662:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760717.501666:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760717.501668:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760717.501674:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760717.501680:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760717.501693:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760717.501769:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 22765568<>22769664 00000100:00100000:1.0:1789760717.501807:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760717.501823:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512570112:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760717.568180:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972877 gen 3 00000008:00010000:1.0:1789760717.568233:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760717.568240:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760717.568248:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760717.568268:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760717.568292:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512570112:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760717.641283:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5559, 5559] 00000080:00200000:3.0:1789760717.641295:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760717.641300:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760717.641311:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22773760 i_blocks 45063 00000020:00200000:3.0:1789760717.641314:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760717.641322:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760717.641353:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22773760, count: 4096 00000080:00200000:3.0:1789760717.641359:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22773760, bytes: 4096 00000080:00200000:3.0:1789760717.641372:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760717.641379:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760717.641381:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22773760, 22777856) 0 0 00000080:00200000:3.0:1789760717.641385:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760717.641398:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5560, 5560] 00020000:00200000:3.0:1789760717.641405:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22773760 chunk: [22773760, 22777856) 22777856, 4096 00020000:00200000:3.0:1789760717.641409:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760717.641423:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5560, 5560] 00000008:00010000:3.0:1789760717.641439:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760717.641444:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760717.641455:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22773760, 22777856) append 0 append lockpos 0 00000080:00200000:3.0:1789760717.641468:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22773760=0x15b8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760717.641513:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760717.641517:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22777856 00000020:00010000:3.0:1789760717.641523:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760717.641526:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760717.641529:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760717.641533:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760717.641536:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760717.641543:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760717.641549:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760717.641564:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760717.641655:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 22773760<>22777856 00000100:00100000:3.0:1789760717.695024:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972879 gen 3 00000008:00010000:3.0:1789760717.695069:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760717.695086:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760717.695099:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760717.695117:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760717.695141:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512570368:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760717.695995:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512570496:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760717.743889:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972880 gen 3 00000008:00010000:1.0:1789760717.743957:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760717.743964:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760717.743974:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760717.744001:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760717.744024:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512570496:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760717.744314:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5561, 5561] 00000080:00200000:3.0:1789760717.744326:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760717.744332:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760717.744343:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22781952 i_blocks 45063 00000020:00200000:3.0:1789760717.744347:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760717.744355:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760717.744380:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22781952, count: 4096 00000080:00200000:3.0:1789760717.744386:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22781952, bytes: 4096 00000080:00200000:3.0:1789760717.744395:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760717.744402:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760717.744404:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22781952, 22786048) 0 0 00000080:00200000:3.0:1789760717.744408:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760717.744422:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5562, 5562] 00020000:00200000:3.0:1789760717.744430:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22781952 chunk: [22781952, 22786048) 22786048, 4096 00020000:00200000:3.0:1789760717.744434:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760717.744451:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5562, 5562] 00000008:00010000:3.0:1789760717.744477:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760717.744481:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760717.744490:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22781952, 22786048) append 0 append lockpos 0 00000080:00200000:3.0:1789760717.744500:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22781952=0x15ba000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760717.744544:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760717.744547:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22786048 00000020:00010000:3.0:1789760717.744553:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760717.744556:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760717.744559:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760717.744562:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760717.744565:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760717.744571:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760717.744577:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760717.744591:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760717.745079:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 22781952<>22786048 00000100:00100000:3.0:1789760717.806853:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972881 gen 3 00000008:00010000:3.0:1789760717.806891:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760717.806923:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760717.806933:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760717.806952:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760717.806973:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512570624:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760717.807681:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512570752:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760717.881719:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972882 gen 3 00000008:00010000:3.0:1789760717.881756:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760717.881763:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760717.881773:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760717.881795:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760717.881818:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512570752:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760717.882335:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760717.882349:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512570880:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760717.927923:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760717.927936:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760717.927946:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760717.928640:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760717.928657:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512570880:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760717.965753:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5565, 5565] 00000080:00200000:3.0:1789760717.965765:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760717.965770:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760717.965782:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22798336 i_blocks 45063 00000020:00200000:3.0:1789760717.965785:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760717.965793:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760717.965829:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22798336, count: 4096 00000080:00200000:3.0:1789760717.965836:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22798336, bytes: 4096 00000080:00200000:3.0:1789760717.965845:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760717.965851:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760717.965853:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22798336, 22802432) 0 0 00000080:00200000:3.0:1789760717.965857:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760717.965869:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5566, 5566] 00020000:00200000:3.0:1789760717.965876:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22798336 chunk: [22798336, 22802432) 22802432, 4096 00020000:00200000:3.0:1789760717.965880:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760717.965894:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5566, 5566] 00000008:00010000:3.0:1789760717.965933:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760717.965938:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760717.965947:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22798336, 22802432) append 0 append lockpos 0 00000080:00200000:3.0:1789760717.965958:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22798336=0x15be000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760717.966002:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760717.966004:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22802432 00000020:00010000:3.0:1789760717.966011:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760717.966014:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760717.966017:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760717.966020:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760717.966022:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760717.966028:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760717.966033:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760717.966178:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760717.966256:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 22798336<>22802432 00000100:00100000:3.0:1789760717.966294:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760717.966307:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512571136:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760718.037229:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972885 gen 3 00000008:00010000:3.0:1789760718.037262:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760718.037269:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760718.037286:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760718.037346:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760718.037366:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512571136:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760718.038947:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512571264:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760718.081449:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760718.081459:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760718.081468:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760718.081487:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760718.081509:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512571264:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760718.081593:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5567, 5567] 00000080:00200000:3.0:1789760718.081612:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760718.081617:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760718.081630:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22806528 i_blocks 45063 00000020:00200000:3.0:1789760718.081633:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760718.081640:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760718.081665:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22806528, count: 4096 00000080:00200000:3.0:1789760718.081672:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22806528, bytes: 4096 00000080:00200000:3.0:1789760718.081680:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760718.081687:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760718.081689:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22806528, 22810624) 0 0 00000080:00200000:3.0:1789760718.081693:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760718.081716:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5568, 5568] 00020000:00200000:3.0:1789760718.081723:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22806528 chunk: [22806528, 22810624) 22810624, 4096 00020000:00200000:3.0:1789760718.081726:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760718.081740:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5568, 5568] 00000008:00010000:3.0:1789760718.081754:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760718.081758:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760718.081767:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22806528, 22810624) append 0 append lockpos 0 00000080:00200000:3.0:1789760718.081778:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22806528=0x15c0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760718.081824:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760718.081826:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22810624 00000020:00010000:3.0:1789760718.081832:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760718.081836:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760718.081838:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760718.081841:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760718.081844:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760718.081849:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760718.081855:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760718.081868:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760718.081982:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 22806528<>22810624 00000100:00100000:3.0:1789760718.130459:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972887 gen 3 00000008:00010000:3.0:1789760718.130504:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760718.130511:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760718.130519:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760718.130538:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760718.130559:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512571392:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760718.131472:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512571520:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760718.188838:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972888 gen 3 00000008:00010000:1.0:1789760718.188872:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760718.188877:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760718.188886:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760718.188936:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760718.188959:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512571520:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760718.192335:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760718.192360:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512571648:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760718.266039:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972889 gen 3 00000008:00010000:1.0:1789760718.266076:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760718.266082:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760718.266092:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760718.266111:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760718.266135:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512571648:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760718.266860:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512571776:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760718.307271:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972890 gen 3 00000008:00010000:3.0:1789760718.307309:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760718.307316:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760718.307324:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760718.307341:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760718.307365:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512571776:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760718.307933:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760718.307947:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512571904:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760718.373253:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760718.373265:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760718.373276:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760718.373301:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760718.373327:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512571904:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760718.458810:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972892 gen 3 00000008:00010000:3.0:1789760718.458857:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760718.458863:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760718.458871:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760718.458887:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760718.458952:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512572032:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760718.459792:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760718.459809:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512572160:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760718.519269:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972893 gen 3 00000008:00010000:3.0:1789760718.519316:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760718.519327:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760718.519338:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760718.519354:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760718.519381:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512572160:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760718.520252:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760718.520271:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512572288:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760718.579291:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972894 gen 3 00000008:00010000:3.0:1789760718.579342:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760718.579350:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760718.579361:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760718.579379:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760718.579407:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512572288:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760718.580392:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760718.580416:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512572416:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760718.621148:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512572672:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760718.621282:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1789760718.621290:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e6cf7267-9b19-462a-a263-14115df15dab:2360:1876699512572544:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760718.621321:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2360:1876699512572800:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760718.624044:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e6cf7267-9b19-462a-a263-14115df15dab:2360:1876699512572544:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760718.624071:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2360:1876699512572800:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760718.624083:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512572672:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760718.642424:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972895 gen 3 00000008:00010000:3.0:1789760718.642477:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760718.642486:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760718.642497:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760718.642517:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760718.642555:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512572416:192.168.203.156@tcp:4:dd.0 00000020:00010000:2.0:1789760718.643330:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760718.643334:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760718.643342:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760718.643349:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760718.644255:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760718.644374:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 22843392<>22847488 00000100:00100000:3.0:1789760718.644516:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760718.644543:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512572928:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760718.707435:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972896 gen 3 00000008:00010000:3.0:1789760718.707488:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760718.707496:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760718.707508:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760718.707649:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760718.707683:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512572928:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760718.707797:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5577, 5577] 00000080:00200000:2.0:1789760718.707813:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760718.707820:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760718.707835:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22847488 i_blocks 45063 00000020:00200000:2.0:1789760718.707839:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760718.707849:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760718.707883:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22847488, count: 4096 00000080:00200000:2.0:1789760718.707891:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22847488, bytes: 4096 00000080:00200000:2.0:1789760718.708058:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760718.708068:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760718.708071:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22847488, 22851584) 0 0 00000080:00200000:2.0:1789760718.708076:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760718.708093:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5578, 5578] 00020000:00200000:2.0:1789760718.708102:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22847488 chunk: [22847488, 22851584) 22851584, 4096 00020000:00200000:2.0:1789760718.708107:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760718.708127:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5578, 5578] 00000008:00010000:2.0:1789760718.708144:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760718.708150:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760718.708162:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22847488, 22851584) append 0 append lockpos 0 00000080:00200000:2.0:1789760718.708175:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22847488=0x15ca000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760718.708233:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760718.708237:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22851584 00000020:00010000:2.0:1789760718.708245:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760718.708249:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760718.708252:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760718.708255:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760718.708259:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760718.708267:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760718.708274:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760718.708291:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760718.708394:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 22847488<>22851584 00000100:00100000:3.0:1789760718.708454:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512573056:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760718.817109:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972897 gen 3 00000008:00010000:3.0:1789760718.817169:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760718.817179:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760718.817191:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760718.817376:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760718.817413:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512573056:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760718.817515:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5578, 5578] 00000080:00200000:2.0:1789760718.817531:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760718.817538:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760718.817666:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22851584 i_blocks 45063 00000020:00200000:2.0:1789760718.817671:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760718.817682:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760718.817720:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22851584, count: 4096 00000080:00200000:2.0:1789760718.817728:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22851584, bytes: 4096 00000080:00200000:2.0:1789760718.817743:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760718.817754:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760718.817756:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22851584, 22855680) 0 0 00000080:00200000:2.0:1789760718.817761:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760718.817783:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5579, 5579] 00020000:00200000:2.0:1789760718.817856:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22851584 chunk: [22851584, 22855680) 22855680, 4096 00020000:00200000:2.0:1789760718.817862:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760718.817885:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5579, 5579] 00000008:00010000:2.0:1789760718.818055:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760718.818062:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760718.818075:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22851584, 22855680) append 0 append lockpos 0 00000080:00200000:2.0:1789760718.818090:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22851584=0x15cb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760718.818153:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760718.818156:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22855680 00000020:00010000:2.0:1789760718.818166:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760718.818170:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760718.818172:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760718.818176:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760718.818180:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760718.818187:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760718.818194:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760718.818215:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760718.818455:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 22851584<>22855680 00000100:00100000:3.0:1789760718.818538:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512573184:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760718.873884:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972898 gen 3 00000008:00010000:3.0:1789760718.873978:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760718.873987:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760718.873998:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760718.874019:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760718.874048:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512573184:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760718.874445:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5579, 5579] 00000080:00200000:2.0:1789760718.874465:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760718.874471:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760718.874485:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22855680 i_blocks 45063 00000020:00200000:2.0:1789760718.874489:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760718.874499:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760718.874533:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22855680, count: 4096 00000080:00200000:2.0:1789760718.874540:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22855680, bytes: 4096 00000080:00200000:2.0:1789760718.874549:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760718.876875:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760718.876881:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22855680, 22859776) 0 0 00000080:00200000:2.0:1789760718.876888:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760718.876920:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5580, 5580] 00020000:00200000:2.0:1789760718.876931:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22855680 chunk: [22855680, 22859776) 22859776, 4096 00020000:00200000:2.0:1789760718.876936:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760718.876962:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5580, 5580] 00000008:00010000:2.0:1789760718.876985:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760718.876991:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760718.877004:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22855680, 22859776) append 0 append lockpos 0 00000080:00200000:2.0:1789760718.877023:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22855680=0x15cc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760718.877091:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760718.877094:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22859776 00000020:00010000:2.0:1789760718.877102:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760718.877105:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760718.877108:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760718.877111:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760718.877115:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760718.877126:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760718.877133:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760718.877160:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760718.877267:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 22855680<>22859776 00000100:00100000:3.0:1789760718.879748:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512573312:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760718.948200:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972899 gen 3 00000008:00010000:3.0:1789760718.948254:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760718.948263:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760718.948274:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760718.948352:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760718.948384:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512573312:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760718.948771:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5580, 5580] 00000080:00200000:2.0:1789760718.948789:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760718.948795:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760718.948811:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22859776 i_blocks 45063 00000020:00200000:2.0:1789760718.948814:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760718.948825:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760718.948859:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22859776, count: 4096 00000080:00200000:2.0:1789760718.948866:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22859776, bytes: 4096 00000080:00200000:2.0:1789760718.948880:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760718.948888:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760718.948891:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22859776, 22863872) 0 0 00000080:00200000:2.0:1789760718.948895:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760718.948989:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5581, 5581] 00020000:00200000:2.0:1789760718.948998:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22859776 chunk: [22859776, 22863872) 22863872, 4096 00020000:00200000:2.0:1789760718.949145:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760718.949170:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5581, 5581] 00000008:00010000:2.0:1789760718.949189:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760718.949195:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760718.949208:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22859776, 22863872) append 0 append lockpos 0 00000080:00200000:2.0:1789760718.949222:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22859776=0x15cd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760718.949282:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760718.949285:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22863872 00000020:00010000:2.0:1789760718.949295:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760718.949298:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760718.949301:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760718.949304:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760718.949308:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760718.949316:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760718.949323:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760718.949340:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760718.949443:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 22859776<>22863872 00000100:00100000:3.0:1789760718.949629:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512573440:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760719.010684:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972900 gen 3 00000008:00010000:3.0:1789760719.010736:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760719.010745:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760719.010758:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760719.010780:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760719.010812:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512573440:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760719.010989:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5581, 5581] 00000080:00200000:2.0:1789760719.011006:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760719.011012:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760719.011028:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22863872 i_blocks 45063 00000020:00200000:2.0:1789760719.011032:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760719.011042:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760719.011081:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22863872, count: 4096 00000080:00200000:2.0:1789760719.011088:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22863872, bytes: 4096 00000080:00200000:2.0:1789760719.011098:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760719.011107:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760719.011109:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22863872, 22867968) 0 0 00000080:00200000:2.0:1789760719.011114:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760719.011132:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5582, 5582] 00020000:00200000:2.0:1789760719.011141:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22863872 chunk: [22863872, 22867968) 22867968, 4096 00020000:00200000:2.0:1789760719.011147:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760719.011165:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5582, 5582] 00000008:00010000:2.0:1789760719.011183:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760719.011189:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760719.011200:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22863872, 22867968) append 0 append lockpos 0 00000080:00200000:2.0:1789760719.011215:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22863872=0x15ce000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760719.011272:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760719.011274:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22867968 00000020:00010000:2.0:1789760719.011283:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760719.011286:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760719.011289:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760719.011292:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760719.011296:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760719.011304:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760719.011310:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760719.011326:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760719.011428:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 22863872<>22867968 00000100:00100000:3.0:1789760719.011661:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512573568:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760719.075996:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972901 gen 3 00000008:00010000:3.0:1789760719.076054:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760719.076063:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760719.076073:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760719.076095:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760719.076126:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512573568:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760719.076403:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5582, 5582] 00000080:00200000:2.0:1789760719.076419:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760719.076426:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760719.076441:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22867968 i_blocks 45063 00000020:00200000:2.0:1789760719.076445:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760719.076455:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760719.076489:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22867968, count: 4096 00000080:00200000:2.0:1789760719.076495:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22867968, bytes: 4096 00000080:00200000:2.0:1789760719.076507:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760719.076529:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760719.076531:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22867968, 22872064) 0 0 00000080:00200000:2.0:1789760719.076535:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760719.076551:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5583, 5583] 00020000:00200000:2.0:1789760719.076559:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22867968 chunk: [22867968, 22872064) 22872064, 4096 00020000:00200000:2.0:1789760719.076564:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760719.076586:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5583, 5583] 00000008:00010000:2.0:1789760719.076614:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760719.076620:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760719.076633:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22867968, 22872064) append 0 append lockpos 0 00000080:00200000:2.0:1789760719.076648:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22867968=0x15cf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760719.076715:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760719.076718:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22872064 00000020:00010000:2.0:1789760719.076726:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760719.076729:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760719.076732:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760719.076736:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760719.076741:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760719.076750:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760719.076758:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760719.076775:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760719.076884:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 22867968<>22872064 00000100:00100000:3.0:1789760719.078424:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512573696:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760719.163946:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972902 gen 3 00000008:00010000:3.0:1789760719.164006:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760719.164014:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760719.164026:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760719.164048:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760719.164083:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512573696:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760719.164441:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5583, 5583] 00000080:00200000:2.0:1789760719.164461:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760719.164467:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760719.164484:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22872064 i_blocks 45063 00000020:00200000:2.0:1789760719.164488:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760719.164499:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760719.164544:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22872064, count: 4096 00000080:00200000:2.0:1789760719.164551:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22872064, bytes: 4096 00000080:00200000:2.0:1789760719.164566:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760719.164575:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760719.164577:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22872064, 22876160) 0 0 00000080:00200000:2.0:1789760719.164582:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760719.164608:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5584, 5584] 00020000:00200000:2.0:1789760719.164617:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22872064 chunk: [22872064, 22876160) 22876160, 4096 00020000:00200000:2.0:1789760719.164623:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760719.164643:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5584, 5584] 00000008:00010000:2.0:1789760719.164661:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760719.164667:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760719.164679:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22872064, 22876160) append 0 append lockpos 0 00000080:00200000:2.0:1789760719.164705:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22872064=0x15d0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760719.164759:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760719.164761:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22876160 00000020:00010000:2.0:1789760719.164769:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760719.164772:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760719.164775:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760719.164778:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760719.164781:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760719.164788:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760719.164795:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760719.164814:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000008:00100000:2.0:1789760719.164958:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 22872064<>22876160 00000100:00100000:3.0:1789760719.165280:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512573824:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760719.250380:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760719.250396:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760719.250410:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760719.250481:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760719.250514:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512573824:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760719.250696:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5584, 5584] 00000080:00200000:2.0:1789760719.250712:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760719.250719:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760719.250932:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22876160 i_blocks 45063 00000020:00200000:3.0:1789760719.250937:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760719.250948:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760719.251042:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22876160, count: 4096 00000080:00200000:3.0:1789760719.251050:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22876160, bytes: 4096 00000080:00200000:3.0:1789760719.251063:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760719.251072:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760719.251074:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22876160, 22880256) 0 0 00000080:00200000:3.0:1789760719.251080:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760719.251097:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5585, 5585] 00020000:00200000:3.0:1789760719.251106:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22876160 chunk: [22876160, 22880256) 22880256, 4096 00020000:00200000:3.0:1789760719.251111:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760719.251133:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5585, 5585] 00000008:00010000:3.0:1789760719.251152:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760719.251158:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760719.251170:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22876160, 22880256) append 0 append lockpos 0 00000080:00200000:3.0:1789760719.251183:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22876160=0x15d1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760719.251365:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760719.251368:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22880256 00000020:00010000:3.0:1789760719.251378:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760719.251381:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760719.251386:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42518/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760719.251389:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760719.251392:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760719.251399:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760719.251406:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760719.251419:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760719.251526:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 22876160<>22880256 00000100:00100000:2.0:1789760719.251612:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512573952:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760719.363965:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972904 gen 3 00000008:00010000:3.0:1789760719.364044:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760719.364054:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760719.364065:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760719.364095:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760719.364125:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512573952:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760719.364971:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512574080:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760719.419608:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972905 gen 3 00000008:00010000:1.0:1789760719.419645:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760719.419651:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760719.419661:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760719.419686:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760719.419719:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512574080:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760719.419843:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5586, 5586] 00000080:00200000:3.0:1789760719.419858:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760719.419864:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760719.419878:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22884352 i_blocks 45063 00000020:00200000:3.0:1789760719.419881:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760719.419890:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760719.419940:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22884352, count: 4096 00000080:00200000:3.0:1789760719.419947:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22884352, bytes: 4096 00000080:00200000:3.0:1789760719.419957:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760719.419965:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760719.419967:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22884352, 22888448) 0 0 00000080:00200000:3.0:1789760719.419971:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760719.419982:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5587, 5587] 00020000:00200000:3.0:1789760719.419988:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22884352 chunk: [22884352, 22888448) 22888448, 4096 00020000:00200000:3.0:1789760719.419992:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760719.420007:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5587, 5587] 00000008:00010000:3.0:1789760719.420020:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760719.420025:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760719.420034:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22884352, 22888448) append 0 append lockpos 0 00000080:00200000:3.0:1789760719.420046:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22884352=0x15d3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760719.420099:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760719.420102:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22888448 00000020:00010000:3.0:1789760719.420110:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760719.420113:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760719.420116:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42518/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760719.420119:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760719.420121:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760719.420127:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760719.420133:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760719.420147:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760719.420251:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 22884352<>22888448 00000100:00100000:3.0:1789760719.420293:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760719.420307:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512574208:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760719.461246:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760719.461261:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760719.461273:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760719.461328:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760719.461360:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512574208:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760719.461433:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5587, 5587] 00000080:00200000:3.0:1789760719.461450:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760719.461456:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760719.461545:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22888448 i_blocks 45063 00000020:00200000:2.0:1789760719.461550:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760719.461559:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760719.461604:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22888448, count: 4096 00000080:00200000:2.0:1789760719.461611:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22888448, bytes: 4096 00000080:00200000:2.0:1789760719.461622:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760719.461631:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760719.461633:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22888448, 22892544) 0 0 00000080:00200000:2.0:1789760719.461637:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760719.461653:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5588, 5588] 00020000:00200000:2.0:1789760719.461661:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22888448 chunk: [22888448, 22892544) 22892544, 4096 00020000:00200000:2.0:1789760719.461665:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760719.461682:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5588, 5588] 00000008:00010000:2.0:1789760719.461698:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760719.461703:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760719.461714:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22888448, 22892544) append 0 append lockpos 0 00000080:00200000:2.0:1789760719.461728:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22888448=0x15d4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760719.461785:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760719.461893:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22892544 00000020:00010000:2.0:1789760719.461926:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760719.461930:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760719.461933:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760719.461937:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760719.461940:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760719.461948:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760719.461956:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760719.461971:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760719.462075:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 22888448<>22892544 00000100:00100000:3.0:1789760719.462139:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512574336:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760719.501469:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972907 gen 3 00000008:00010000:3.0:1789760719.501599:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760719.501607:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760719.501618:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760719.501700:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760719.501732:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512574336:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760719.501928:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5588, 5588] 00000080:00200000:2.0:1789760719.501946:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760719.501952:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760719.501966:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22892544 i_blocks 45063 00000020:00200000:2.0:1789760719.501970:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760719.502094:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760719.502132:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22892544, count: 4096 00000080:00200000:2.0:1789760719.502139:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22892544, bytes: 4096 00000080:00200000:2.0:1789760719.502149:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760719.502158:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760719.502160:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22892544, 22896640) 0 0 00000080:00200000:2.0:1789760719.502165:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760719.502182:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5589, 5589] 00020000:00200000:2.0:1789760719.502190:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22892544 chunk: [22892544, 22896640) 22896640, 4096 00020000:00200000:2.0:1789760719.502194:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760719.502214:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5589, 5589] 00000008:00010000:2.0:1789760719.502231:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760719.502237:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760719.502248:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22892544, 22896640) append 0 append lockpos 0 00000080:00200000:2.0:1789760719.502395:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22892544=0x15d5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760719.502455:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760719.502459:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22896640 00000020:00010000:2.0:1789760719.502467:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760719.502470:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760719.502474:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760719.502477:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760719.502480:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760719.502488:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760719.502494:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760719.502520:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760719.502644:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 22892544<>22896640 00000100:00100000:3.0:1789760719.502806:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512574464:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760719.535954:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972908 gen 3 00000008:00010000:2.0:1789760719.536072:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:2.0:1789760719.536080:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760719.536091:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760719.536111:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:2.0:1789760719.536140:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512574464:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760719.536837:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760719.536869:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512574592:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760719.587515:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972909 gen 3 00000008:00010000:1.0:1789760719.587564:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760719.587572:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760719.587584:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760719.587676:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760719.587709:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512574592:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760719.588317:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512574720:192.168.203.156@tcp:4:dd.0 00010000:00010000:3.0:1789760719.645605:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760719.663618:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972910 gen 3 00000008:00010000:1.0:1789760719.663655:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760719.663661:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760719.663670:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760719.663813:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760719.663836:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512574720:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760719.668424:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512574848:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760719.788382:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972911 gen 3 00000008:00010000:1.0:1789760719.788420:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760719.788427:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760719.788437:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760719.788461:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760719.788483:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512574848:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760719.789100:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512574976:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760719.853182:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972912 gen 3 00000008:00010000:1.0:1789760719.853501:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760719.853511:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760719.853524:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760719.853550:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760719.853597:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512574976:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760719.854644:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512575104:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760719.915307:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760719.915324:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760719.915338:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760719.915365:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760719.915510:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512575104:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760720.012360:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5595, 5595] 00000080:00200000:1.0:1789760720.012373:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760720.012378:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760720.012391:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22921216 i_blocks 45063 00000020:00200000:1.0:1789760720.012396:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760720.012404:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760720.012436:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22921216, count: 4096 00000080:00200000:1.0:1789760720.012442:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22921216, bytes: 4096 00000080:00200000:1.0:1789760720.012452:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760720.012460:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760720.012462:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22921216, 22925312) 0 0 00000080:00200000:1.0:1789760720.012468:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760720.012484:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5596, 5596] 00020000:00200000:1.0:1789760720.012493:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22921216 chunk: [22921216, 22925312) 22925312, 4096 00020000:00200000:1.0:1789760720.012497:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760720.012514:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5596, 5596] 00000008:00010000:1.0:1789760720.012529:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760720.012536:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760720.012544:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22921216, 22925312) append 0 append lockpos 0 00000080:00200000:1.0:1789760720.012558:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22921216=0x15dc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760720.012615:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760720.012618:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22925312 00000020:00010000:1.0:1789760720.012624:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760720.012628:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.012630:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.012633:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760720.012636:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760720.012641:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760720.012647:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760720.012661:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760720.012736:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 22921216<>22925312 00000100:00100000:1.0:1789760720.013261:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760720.013281:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512575360:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760720.046704:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972915 gen 3 00000008:00010000:1.0:1789760720.046740:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760720.046746:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760720.046754:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760720.046824:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760720.046848:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512575360:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760720.046989:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5596, 5596] 00000080:00200000:2.0:1789760720.047002:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760720.047007:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760720.047017:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22925312 i_blocks 45063 00000020:00200000:2.0:1789760720.047021:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760720.047028:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760720.047055:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22925312, count: 4096 00000080:00200000:2.0:1789760720.047061:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22925312, bytes: 4096 00000080:00200000:2.0:1789760720.047069:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760720.047076:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760720.047078:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22925312, 22929408) 0 0 00000080:00200000:2.0:1789760720.047082:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760720.047094:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5597, 5597] 00020000:00200000:2.0:1789760720.047956:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22925312 chunk: [22925312, 22929408) 22929408, 4096 00020000:00200000:2.0:1789760720.047962:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760720.047990:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5597, 5597] 00000008:00010000:2.0:1789760720.048005:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760720.048010:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760720.048019:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22925312, 22929408) append 0 append lockpos 0 00000080:00200000:2.0:1789760720.048030:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22925312=0x15dd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760720.048074:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760720.048077:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22929408 00000020:00010000:2.0:1789760720.048083:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760720.048086:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760720.048089:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760720.048092:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760720.048094:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760720.048509:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760720.048517:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760720.048537:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760720.048622:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 22925312<>22929408 00000100:00100000:3.0:1789760720.048936:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512575488:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760720.082339:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760720.082350:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760720.082359:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760720.082368:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760720.082393:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512575488:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760720.123388:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5598, 5598] 00000080:00200000:1.0:1789760720.123401:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760720.123405:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760720.123417:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22933504 i_blocks 45063 00000020:00200000:1.0:1789760720.123421:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760720.123428:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760720.123454:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22933504, count: 4096 00000080:00200000:1.0:1789760720.123459:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22933504, bytes: 4096 00000080:00200000:1.0:1789760720.123466:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760720.123473:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760720.123475:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22933504, 22937600) 0 0 00000080:00200000:1.0:1789760720.123478:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760720.123492:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5599, 5599] 00020000:00200000:1.0:1789760720.123497:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22933504 chunk: [22933504, 22937600) 22937600, 4096 00020000:00200000:1.0:1789760720.123501:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760720.123516:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5599, 5599] 00000008:00010000:1.0:1789760720.123530:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760720.123535:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760720.123544:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22933504, 22937600) append 0 append lockpos 0 00000080:00200000:1.0:1789760720.123555:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22933504=0x15df000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760720.123614:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760720.123617:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22937600 00000020:00010000:1.0:1789760720.123624:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760720.123628:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.123630:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.123634:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760720.123636:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760720.123642:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760720.123648:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760720.123660:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760720.123739:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 22933504<>22937600 00000080:00200000:1.0:1789760720.175287:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5599, 5599] 00000080:00200000:1.0:1789760720.175300:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760720.175305:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760720.175317:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22937600 i_blocks 45063 00000020:00200000:1.0:1789760720.175321:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760720.175328:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760720.175352:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22937600, count: 4096 00000080:00200000:1.0:1789760720.175359:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22937600, bytes: 4096 00000080:00200000:1.0:1789760720.175366:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760720.175373:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760720.175375:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22937600, 22941696) 0 0 00000080:00200000:1.0:1789760720.175379:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760720.175394:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5600, 5600] 00020000:00200000:1.0:1789760720.175402:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22937600 chunk: [22937600, 22941696) 22941696, 4096 00020000:00200000:1.0:1789760720.175406:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760720.175420:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5600, 5600] 00000008:00010000:1.0:1789760720.175435:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760720.175439:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760720.175448:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22937600, 22941696) append 0 append lockpos 0 00000080:00200000:1.0:1789760720.175459:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22937600=0x15e0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760720.175503:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760720.175505:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22941696 00000020:00010000:1.0:1789760720.175512:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760720.175515:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.175518:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.175520:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760720.175523:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760720.175529:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760720.175535:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760720.175549:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760720.175636:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96680 - object 0x240000400:5 offset 22937600<>22941696 00000080:00200000:1.0:1789760720.209572:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5600, 5600] 00000080:00200000:1.0:1789760720.209594:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760720.209600:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760720.209611:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22941696 i_blocks 45063 00000020:00200000:1.0:1789760720.209614:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760720.209622:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760720.209647:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22941696, count: 4096 00000080:00200000:1.0:1789760720.209653:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22941696, bytes: 4096 00000080:00200000:1.0:1789760720.209661:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760720.209667:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760720.209670:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22941696, 22945792) 0 0 00000080:00200000:1.0:1789760720.209673:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760720.209685:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5601, 5601] 00020000:00200000:1.0:1789760720.209691:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22941696 chunk: [22941696, 22945792) 22945792, 4096 00020000:00200000:1.0:1789760720.209695:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760720.209709:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5601, 5601] 00000008:00010000:1.0:1789760720.209723:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760720.209727:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760720.209736:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22941696, 22945792) append 0 append lockpos 0 00000080:00200000:1.0:1789760720.209746:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22941696=0x15e1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760720.209789:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760720.209792:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22945792 00000020:00010000:1.0:1789760720.209799:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760720.209802:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.209805:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.209808:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760720.209811:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760720.209817:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760720.209823:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760720.209837:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f429c0 00000008:00100000:1.0:1789760720.209953:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97480 - object 0x240000400:5 offset 22941696<>22945792 00000080:00200000:1.0:1789760720.298294:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5601, 5601] 00000080:00200000:1.0:1789760720.298308:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760720.298314:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760720.298325:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22945792 i_blocks 45063 00000020:00200000:1.0:1789760720.298329:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760720.298338:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760720.298369:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22945792, count: 4096 00000080:00200000:1.0:1789760720.298376:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22945792, bytes: 4096 00000080:00200000:1.0:1789760720.298385:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760720.298392:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760720.298394:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22945792, 22949888) 0 0 00000080:00200000:1.0:1789760720.298399:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760720.298415:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5602, 5602] 00020000:00200000:1.0:1789760720.298422:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22945792 chunk: [22945792, 22949888) 22949888, 4096 00020000:00200000:1.0:1789760720.298426:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760720.298442:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5602, 5602] 00000008:00010000:1.0:1789760720.298461:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760720.298466:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760720.298476:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22945792, 22949888) append 0 append lockpos 0 00000080:00200000:1.0:1789760720.298487:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22945792=0x15e2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760720.298535:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760720.298538:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22949888 00000020:00010000:1.0:1789760720.298545:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760720.298548:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.298551:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.298554:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760720.298557:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760720.298563:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760720.298568:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760720.298593:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42900 00000008:00100000:1.0:1789760720.298679:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95880 - object 0x240000400:5 offset 22945792<>22949888 00000080:00200000:1.0:1789760720.376269:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5602, 5602] 00000080:00200000:1.0:1789760720.376292:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760720.376298:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760720.376312:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22949888 i_blocks 45063 00000020:00200000:1.0:1789760720.376315:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760720.376324:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760720.376348:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22949888, count: 4096 00000080:00200000:1.0:1789760720.376354:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22949888, bytes: 4096 00000080:00200000:1.0:1789760720.376363:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760720.376370:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760720.376372:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22949888, 22953984) 0 0 00000080:00200000:1.0:1789760720.376376:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760720.376390:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5603, 5603] 00020000:00200000:1.0:1789760720.376398:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22949888 chunk: [22949888, 22953984) 22953984, 4096 00020000:00200000:1.0:1789760720.376402:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760720.376419:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5603, 5603] 00000008:00010000:1.0:1789760720.376436:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760720.376441:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760720.376451:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22949888, 22953984) append 0 append lockpos 0 00000080:00200000:1.0:1789760720.376462:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22949888=0x15e3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760720.376510:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760720.376513:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22953984 00000020:00010000:1.0:1789760720.376519:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760720.376523:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.376526:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.376529:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760720.376531:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760720.376538:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760720.376543:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760720.376557:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:1.0:1789760720.376643:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96d80 - object 0x240000400:5 offset 22949888<>22953984 00000080:00200000:1.0:1789760720.476851:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5603, 5603] 00000080:00200000:1.0:1789760720.476865:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760720.476869:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760720.476881:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22953984 i_blocks 45063 00000020:00200000:1.0:1789760720.476884:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760720.476892:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760720.476944:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22953984, count: 4096 00000080:00200000:1.0:1789760720.476951:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22953984, bytes: 4096 00000080:00200000:1.0:1789760720.476960:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760720.476968:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760720.476975:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22953984, 22958080) 0 0 00000080:00200000:1.0:1789760720.476979:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760720.476992:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5604, 5604] 00020000:00200000:1.0:1789760720.476998:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22953984 chunk: [22953984, 22958080) 22958080, 4096 00020000:00200000:1.0:1789760720.477003:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760720.477019:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5604, 5604] 00000008:00010000:1.0:1789760720.477034:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760720.477039:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760720.477050:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22953984, 22958080) append 0 append lockpos 0 00000080:00200000:1.0:1789760720.477060:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22953984=0x15e4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760720.477105:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760720.477108:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22958080 00000020:00010000:1.0:1789760720.477114:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760720.477118:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.477121:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.477124:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760720.477127:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760720.477134:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760720.477139:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760720.477153:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760720.478114:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96d80 - object 0x240000400:5 offset 22953984<>22958080 00000080:00200000:1.0:1789760720.558924:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5604, 5604] 00000080:00200000:1.0:1789760720.558940:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760720.558946:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760720.558967:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22958080 i_blocks 45063 00000020:00200000:1.0:1789760720.558971:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760720.558979:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760720.559007:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22958080, count: 4096 00000080:00200000:1.0:1789760720.559013:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22958080, bytes: 4096 00000080:00200000:1.0:1789760720.559021:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760720.559029:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760720.559032:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22958080, 22962176) 0 0 00000080:00200000:1.0:1789760720.559035:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760720.559050:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5605, 5605] 00020000:00200000:1.0:1789760720.559059:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22958080 chunk: [22958080, 22962176) 22962176, 4096 00020000:00200000:1.0:1789760720.559063:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760720.559079:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5605, 5605] 00000008:00010000:1.0:1789760720.559092:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760720.559097:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760720.559105:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22958080, 22962176) append 0 append lockpos 0 00000080:00200000:1.0:1789760720.559116:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22958080=0x15e5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760720.559154:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760720.559155:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22962176 00000020:00010000:1.0:1789760720.559161:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760720.559163:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.559165:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.559167:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760720.559168:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760720.559173:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760720.559177:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760720.559187:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760720.559249:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95880 - object 0x240000400:5 offset 22958080<>22962176 00000080:00200000:1.0:1789760720.658460:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5605, 5605] 00000080:00200000:1.0:1789760720.658472:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760720.658477:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760720.658489:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22962176 i_blocks 45063 00000020:00200000:1.0:1789760720.658492:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760720.658501:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760720.658526:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22962176, count: 4096 00000080:00200000:1.0:1789760720.658532:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22962176, bytes: 4096 00000080:00200000:1.0:1789760720.658543:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760720.658550:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760720.658552:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22962176, 22966272) 0 0 00000080:00200000:1.0:1789760720.658555:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760720.658569:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5606, 5606] 00020000:00200000:1.0:1789760720.658575:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22962176 chunk: [22962176, 22966272) 22966272, 4096 00020000:00200000:1.0:1789760720.658588:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760720.658604:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5606, 5606] 00000008:00010000:1.0:1789760720.658617:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760720.658622:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760720.658632:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22962176, 22966272) append 0 append lockpos 0 00000080:00200000:1.0:1789760720.658643:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22962176=0x15e6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760720.658754:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760720.658757:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22966272 00000020:00010000:1.0:1789760720.658764:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760720.658767:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.658770:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.658773:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760720.658776:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760720.658782:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760720.658850:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760720.658945:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:1.0:1789760720.659411:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97480 - object 0x240000400:5 offset 22962176<>22966272 00000100:00100000:1.0:1789760720.659446:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760720.659462:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512576640:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760720.729272:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42240 00000020:00200000:1.0:1789760720.729283:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760720.729292:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760720.729311:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42240 00000100:00100000:1.0:1789760720.729333:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512576640:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760720.831635:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5607, 5607] 00000080:00200000:1.0:1789760720.831658:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760720.831663:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760720.831674:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22970368 i_blocks 45063 00000020:00200000:1.0:1789760720.831677:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760720.831685:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760720.831709:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22970368, count: 4096 00000080:00200000:1.0:1789760720.831714:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22970368, bytes: 4096 00000080:00200000:1.0:1789760720.831722:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760720.831729:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760720.831731:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22970368, 22974464) 0 0 00000080:00200000:1.0:1789760720.831735:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760720.831748:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5608, 5608] 00020000:00200000:1.0:1789760720.831755:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22970368 chunk: [22970368, 22974464) 22974464, 4096 00020000:00200000:1.0:1789760720.831759:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760720.831783:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5608, 5608] 00000008:00010000:1.0:1789760720.831804:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760720.831808:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760720.831816:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22970368, 22974464) append 0 append lockpos 0 00000080:00200000:1.0:1789760720.831827:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22970368=0x15e8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760720.831868:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760720.831871:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22974464 00000020:00010000:1.0:1789760720.831877:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760720.831880:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.831883:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.831886:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760720.831888:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760720.831894:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760720.831935:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760720.831949:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:1.0:1789760720.832036:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96680 - object 0x240000400:5 offset 22970368<>22974464 00000100:00100000:1.0:1789760720.832063:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760720.832075:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512576896:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760720.905483:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972927 gen 3 00000008:00010000:1.0:1789760720.905516:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42240 00000020:00200000:1.0:1789760720.905522:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760720.905530:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760720.905613:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42240 00000100:00100000:1.0:1789760720.905636:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512576896:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760720.998994:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5609, 5609] 00000080:00200000:1.0:1789760720.999010:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760720.999017:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760720.999030:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22978560 i_blocks 45063 00000020:00200000:1.0:1789760720.999035:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760720.999046:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760720.999070:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22978560, count: 4096 00000080:00200000:1.0:1789760720.999089:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22978560, bytes: 4096 00000080:00200000:1.0:1789760720.999101:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760720.999109:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760720.999111:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22978560, 22982656) 0 0 00000080:00200000:1.0:1789760720.999116:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760720.999130:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5610, 5610] 00020000:00200000:1.0:1789760720.999137:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22978560 chunk: [22978560, 22982656) 22982656, 4096 00020000:00200000:1.0:1789760720.999141:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760720.999166:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5610, 5610] 00000008:00010000:1.0:1789760720.999181:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760720.999185:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760720.999194:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22978560, 22982656) append 0 append lockpos 0 00000080:00200000:1.0:1789760720.999206:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22978560=0x15ea000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760720.999255:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760720.999257:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22982656 00000020:00010000:1.0:1789760720.999265:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760720.999268:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.999271:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760720.999275:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760720.999277:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760720.999284:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760720.999290:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760720.999305:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:1.0:1789760720.999388:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96680 - object 0x240000400:5 offset 22978560<>22982656 00000100:00100000:1.0:1789760720.999434:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760720.999452:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512577152:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760721.038373:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42240 00000020:00200000:1.0:1789760721.038386:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760721.038395:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760721.038473:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42240 00000100:00100000:1.0:1789760721.038497:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512577152:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760721.039182:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512577280:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760721.077151:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972930 gen 3 00000008:00010000:1.0:1789760721.077204:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760721.077211:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760721.077220:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760721.077284:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760721.077311:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512577280:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760721.077396:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5611, 5611] 00000080:00200000:2.0:1789760721.077412:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760721.077417:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760721.077430:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22986752 i_blocks 45063 00000020:00200000:2.0:1789760721.077435:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760721.077444:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760721.077470:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22986752, count: 4096 00000080:00200000:2.0:1789760721.077477:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22986752, bytes: 4096 00000080:00200000:2.0:1789760721.077485:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760721.077491:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760721.077494:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22986752, 22990848) 0 0 00000080:00200000:2.0:1789760721.077511:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760721.077528:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5612, 5612] 00020000:00200000:2.0:1789760721.077538:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22986752 chunk: [22986752, 22990848) 22990848, 4096 00020000:00200000:2.0:1789760721.077541:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760721.077558:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5612, 5612] 00000008:00010000:2.0:1789760721.077571:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760721.077584:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760721.077595:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22986752, 22990848) append 0 append lockpos 0 00000080:00200000:2.0:1789760721.077606:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22986752=0x15ec000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760721.077659:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760721.077662:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22990848 00000020:00010000:2.0:1789760721.077668:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760721.077672:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760721.077674:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760721.077678:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760721.077681:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760721.077687:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760721.077693:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760721.077706:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000008:00100000:2.0:1789760721.078233:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 22986752<>22990848 00000100:00100000:2.0:1789760721.078284:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512577408:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760721.126790:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972931 gen 3 00000008:00010000:3.0:1789760721.126828:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760721.126834:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760721.126843:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760721.126868:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760721.126893:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512577408:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760721.127019:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5612, 5612] 00000080:00200000:1.0:1789760721.127032:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760721.127038:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760721.127054:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22990848 i_blocks 45063 00000020:00200000:1.0:1789760721.127058:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760721.127066:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760721.127095:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22990848, count: 4096 00000080:00200000:1.0:1789760721.127102:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22990848, bytes: 4096 00000080:00200000:1.0:1789760721.127120:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760721.127127:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760721.127129:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22990848, 22994944) 0 0 00000080:00200000:1.0:1789760721.127133:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760721.127146:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5613, 5613] 00020000:00200000:1.0:1789760721.127153:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22990848 chunk: [22990848, 22994944) 22994944, 4096 00020000:00200000:1.0:1789760721.127157:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760721.127173:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5613, 5613] 00000008:00010000:1.0:1789760721.127186:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760721.127190:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760721.127199:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22990848, 22994944) append 0 append lockpos 0 00000080:00200000:1.0:1789760721.127210:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22990848=0x15ed000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760721.127247:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760721.127250:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22994944 00000020:00010000:1.0:1789760721.127257:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760721.127261:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.127264:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.127267:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760721.127270:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760721.127277:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760721.127283:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760721.127928:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:3.0:1789760721.128018:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 22990848<>22994944 00000100:00100000:3.0:1789760721.128067:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512577536:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760721.174027:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972932 gen 3 00000008:00010000:3.0:1789760721.174070:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42240 00000020:00200000:3.0:1789760721.174078:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760721.174088:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760721.174241:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42240 00000100:00100000:3.0:1789760721.174274:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512577536:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760721.174355:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5613, 5613] 00000080:00200000:1.0:1789760721.174371:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760721.174377:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760721.174393:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22994944 i_blocks 45063 00000020:00200000:1.0:1789760721.174397:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760721.174405:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760721.174434:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22994944, count: 4096 00000080:00200000:1.0:1789760721.174441:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22994944, bytes: 4096 00000080:00200000:1.0:1789760721.174449:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760721.174455:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760721.174457:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22994944, 22999040) 0 0 00000080:00200000:1.0:1789760721.174461:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760721.174475:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5614, 5614] 00020000:00200000:1.0:1789760721.174482:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22994944 chunk: [22994944, 22999040) 22999040, 4096 00020000:00200000:1.0:1789760721.174487:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760721.174500:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5614, 5614] 00000008:00010000:1.0:1789760721.174513:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760721.174517:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760721.174525:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22994944, 22999040) append 0 append lockpos 0 00000080:00200000:1.0:1789760721.174535:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22994944=0x15ee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760721.174592:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760721.174595:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 22999040 00000020:00010000:1.0:1789760721.174601:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760721.174603:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.174606:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.174609:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760721.174611:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760721.174617:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760721.174623:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760721.174636:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42900 00000008:00100000:1.0:1789760721.174721:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96680 - object 0x240000400:5 offset 22994944<>22999040 00000080:00200000:1.0:1789760721.230343:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5614, 5614] 00000080:00200000:1.0:1789760721.230355:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760721.230360:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760721.230371:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 22999040 i_blocks 45063 00000020:00200000:1.0:1789760721.230374:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760721.230381:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760721.230405:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 22999040, count: 4096 00000080:00200000:1.0:1789760721.230411:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 22999040, bytes: 4096 00000080:00200000:1.0:1789760721.230418:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760721.230425:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760721.230428:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [22999040, 23003136) 0 0 00000080:00200000:1.0:1789760721.230431:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760721.230443:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5615, 5615] 00020000:00200000:1.0:1789760721.230450:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 22999040 chunk: [22999040, 23003136) 23003136, 4096 00020000:00200000:1.0:1789760721.230454:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760721.230469:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5615, 5615] 00000008:00010000:1.0:1789760721.230483:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760721.230487:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760721.230496:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [22999040, 23003136) append 0 append lockpos 0 00000080:00200000:1.0:1789760721.230508:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=22999040=0x15ef000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760721.230549:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760721.230552:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23003136 00000020:00010000:1.0:1789760721.230558:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760721.230561:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.230564:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.230567:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760721.230569:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760721.230588:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760721.230758:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760721.230774:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f429c0 00000008:00100000:1.0:1789760721.230846:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 22999040<>23003136 00000080:00200000:1.0:1789760721.281926:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5615, 5615] 00000080:00200000:1.0:1789760721.281940:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760721.281945:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760721.281957:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23003136 i_blocks 45063 00000020:00200000:1.0:1789760721.281961:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760721.281980:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760721.282008:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23003136, count: 4096 00000080:00200000:1.0:1789760721.282014:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23003136, bytes: 4096 00000080:00200000:1.0:1789760721.282021:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760721.282028:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760721.282030:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23003136, 23007232) 0 0 00000080:00200000:1.0:1789760721.282034:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760721.282049:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5616, 5616] 00020000:00200000:1.0:1789760721.282057:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23003136 chunk: [23003136, 23007232) 23007232, 4096 00020000:00200000:1.0:1789760721.282060:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760721.282075:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5616, 5616] 00000008:00010000:1.0:1789760721.282087:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760721.282091:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760721.282101:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23003136, 23007232) append 0 append lockpos 0 00000080:00200000:1.0:1789760721.282111:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23003136=0x15f0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760721.282154:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760721.282158:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23007232 00000020:00010000:1.0:1789760721.282164:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760721.282168:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.282171:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.282174:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760721.282177:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760721.282183:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760721.282188:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760721.282202:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760721.282278:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 23003136<>23007232 00000080:00200000:1.0:1789760721.321008:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5616, 5616] 00000080:00200000:1.0:1789760721.321021:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760721.321025:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760721.321037:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23007232 i_blocks 45063 00000020:00200000:1.0:1789760721.321040:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760721.321048:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760721.321075:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23007232, count: 4096 00000080:00200000:1.0:1789760721.321082:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23007232, bytes: 4096 00000080:00200000:1.0:1789760721.321090:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760721.321097:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760721.321099:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23007232, 23011328) 0 0 00000080:00200000:1.0:1789760721.321103:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760721.321117:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5617, 5617] 00020000:00200000:1.0:1789760721.321123:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23007232 chunk: [23007232, 23011328) 23011328, 4096 00020000:00200000:1.0:1789760721.321127:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760721.321142:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5617, 5617] 00000008:00010000:1.0:1789760721.321155:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760721.321160:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760721.321168:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23007232, 23011328) append 0 append lockpos 0 00000080:00200000:1.0:1789760721.321178:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23007232=0x15f1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760721.321221:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760721.321223:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23011328 00000020:00010000:1.0:1789760721.321230:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760721.321233:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.321236:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.321238:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760721.321241:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760721.321247:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760721.321253:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760721.321267:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760721.321345:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94380 - object 0x240000400:5 offset 23007232<>23011328 00000080:00200000:1.0:1789760721.387856:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5617, 5617] 00000080:00200000:1.0:1789760721.387869:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760721.387874:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760721.387885:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23011328 i_blocks 45063 00000020:00200000:1.0:1789760721.387888:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760721.387897:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760721.387948:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23011328, count: 4096 00000080:00200000:1.0:1789760721.387954:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23011328, bytes: 4096 00000080:00200000:1.0:1789760721.387963:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760721.387981:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760721.387983:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23011328, 23015424) 0 0 00000080:00200000:1.0:1789760721.387987:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760721.388000:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5618, 5618] 00020000:00200000:1.0:1789760721.388007:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23011328 chunk: [23011328, 23015424) 23015424, 4096 00020000:00200000:1.0:1789760721.388010:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760721.388026:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5618, 5618] 00000008:00010000:1.0:1789760721.388040:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760721.388045:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760721.388056:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23011328, 23015424) append 0 append lockpos 0 00000080:00200000:1.0:1789760721.388068:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23011328=0x15f2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760721.388114:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760721.388117:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23015424 00000020:00010000:1.0:1789760721.388124:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760721.388127:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.388130:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.388133:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760721.388136:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760721.388142:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760721.388147:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760721.388162:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f429c0 00000008:00100000:1.0:1789760721.388243:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 23011328<>23015424 00000080:00200000:1.0:1789760721.462407:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5618, 5618] 00000080:00200000:1.0:1789760721.462420:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760721.462425:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760721.462439:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23015424 i_blocks 45063 00000020:00200000:1.0:1789760721.462442:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760721.462461:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760721.462488:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23015424, count: 4096 00000080:00200000:1.0:1789760721.462494:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23015424, bytes: 4096 00000080:00200000:1.0:1789760721.462502:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760721.462509:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760721.462511:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23015424, 23019520) 0 0 00000080:00200000:1.0:1789760721.462514:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760721.462527:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5619, 5619] 00020000:00200000:1.0:1789760721.462533:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23015424 chunk: [23015424, 23019520) 23019520, 4096 00020000:00200000:1.0:1789760721.462538:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760721.462553:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5619, 5619] 00000008:00010000:1.0:1789760721.462567:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760721.462582:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760721.462592:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23015424, 23019520) append 0 append lockpos 0 00000080:00200000:1.0:1789760721.462603:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23015424=0x15f3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760721.462645:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760721.462648:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23019520 00000020:00010000:1.0:1789760721.462655:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760721.462658:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.462661:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.462663:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760721.462666:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760721.462672:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760721.462678:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760721.462692:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42900 00000008:00100000:1.0:1789760721.462766:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 23015424<>23019520 00000100:00100000:1.0:1789760721.462804:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760721.462815:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512578304:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760721.506671:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972938 gen 3 00000008:00010000:1.0:1789760721.506711:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42900 00000020:00200000:1.0:1789760721.506717:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760721.506726:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760721.506756:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42900 00000100:00100000:1.0:1789760721.506783:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512578304:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760721.506792:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5619, 5619] 00000080:00200000:3.0:1789760721.506808:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760721.506821:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760721.506832:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23019520 i_blocks 45063 00000020:00200000:3.0:1789760721.506836:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760721.506845:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760721.506871:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23019520, count: 4096 00000080:00200000:3.0:1789760721.506877:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23019520, bytes: 4096 00000080:00200000:3.0:1789760721.506886:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760721.506892:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760721.506894:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23019520, 23023616) 0 0 00000080:00200000:3.0:1789760721.506898:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760721.506936:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5620, 5620] 00020000:00200000:3.0:1789760721.506943:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23019520 chunk: [23019520, 23023616) 23023616, 4096 00020000:00200000:3.0:1789760721.506946:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760721.506961:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5620, 5620] 00000008:00010000:3.0:1789760721.506975:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760721.506980:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760721.506989:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23019520, 23023616) append 0 append lockpos 0 00000080:00200000:3.0:1789760721.506999:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23019520=0x15f4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760721.507046:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760721.507049:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23023616 00000020:00010000:3.0:1789760721.507055:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760721.507059:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.507061:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.507065:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760721.507068:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760721.507075:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760721.507081:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760721.507095:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760721.507185:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 23019520<>23023616 00000100:00100000:2.0:1789760721.507248:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512578432:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760721.560625:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972939 gen 3 00000008:00010000:3.0:1789760721.560660:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:3.0:1789760721.560666:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760721.560676:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760721.560699:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:3.0:1789760721.560723:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512578432:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760721.561034:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5620, 5620] 00000080:00200000:1.0:1789760721.561046:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760721.561050:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760721.561060:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23023616 i_blocks 45063 00000020:00200000:1.0:1789760721.561063:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760721.561071:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760721.561099:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23023616, count: 4096 00000080:00200000:1.0:1789760721.561106:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23023616, bytes: 4096 00000080:00200000:1.0:1789760721.561114:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760721.561122:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760721.561124:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23023616, 23027712) 0 0 00000080:00200000:1.0:1789760721.561128:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760721.561139:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5621, 5621] 00020000:00200000:1.0:1789760721.561146:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23023616 chunk: [23023616, 23027712) 23027712, 4096 00020000:00200000:1.0:1789760721.561150:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760721.561165:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5621, 5621] 00000008:00010000:1.0:1789760721.561178:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760721.561183:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760721.561193:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23023616, 23027712) append 0 append lockpos 0 00000080:00200000:1.0:1789760721.561204:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23023616=0x15f5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760721.561244:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760721.561247:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23027712 00000020:00010000:1.0:1789760721.561253:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760721.561256:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.561259:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.561262:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760721.561264:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760721.561270:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760721.561275:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760721.561288:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42900 00000008:00100000:1.0:1789760721.561363:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 23023616<>23027712 00000080:00200000:1.0:1789760721.597335:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5621, 5621] 00000080:00200000:1.0:1789760721.597350:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760721.597356:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760721.597368:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23027712 i_blocks 45063 00000020:00200000:1.0:1789760721.597372:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760721.597379:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760721.597405:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23027712, count: 4096 00000080:00200000:1.0:1789760721.597411:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23027712, bytes: 4096 00000080:00200000:1.0:1789760721.597419:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760721.597427:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760721.597429:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23027712, 23031808) 0 0 00000080:00200000:1.0:1789760721.597433:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760721.597445:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5622, 5622] 00020000:00200000:1.0:1789760721.597452:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23027712 chunk: [23027712, 23031808) 23031808, 4096 00020000:00200000:1.0:1789760721.597456:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760721.597481:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5622, 5622] 00000008:00010000:1.0:1789760721.597494:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760721.597499:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760721.597508:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23027712, 23031808) append 0 append lockpos 0 00000080:00200000:1.0:1789760721.597518:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23027712=0x15f6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760721.597563:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760721.597566:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23031808 00000020:00010000:1.0:1789760721.597592:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760721.597596:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.597598:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.597601:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760721.597604:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760721.597609:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760721.597614:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760721.597628:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:1.0:1789760721.597702:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94700 - object 0x240000400:5 offset 23027712<>23031808 00000080:00200000:1.0:1789760721.630051:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5622, 5622] 00000080:00200000:1.0:1789760721.630065:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760721.630082:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760721.630095:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23031808 i_blocks 45063 00000020:00200000:1.0:1789760721.630100:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760721.630109:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760721.630135:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23031808, count: 4096 00000080:00200000:1.0:1789760721.630141:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23031808, bytes: 4096 00000080:00200000:1.0:1789760721.630150:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760721.630168:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760721.630171:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23031808, 23035904) 0 0 00000080:00200000:1.0:1789760721.630174:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760721.630187:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5623, 5623] 00020000:00200000:1.0:1789760721.630193:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23031808 chunk: [23031808, 23035904) 23035904, 4096 00020000:00200000:1.0:1789760721.630197:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760721.630212:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5623, 5623] 00000008:00010000:1.0:1789760721.630226:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760721.630231:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760721.630240:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23031808, 23035904) append 0 append lockpos 0 00000080:00200000:1.0:1789760721.630250:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23031808=0x15f7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760721.630306:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760721.630309:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23035904 00000020:00010000:1.0:1789760721.630316:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760721.630319:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.630322:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.630326:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760721.630328:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760721.630336:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760721.630342:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760721.630361:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760721.630439:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 23031808<>23035904 00000080:00200000:1.0:1789760721.681727:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5623, 5623] 00000080:00200000:1.0:1789760721.681745:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760721.681750:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760721.681763:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23035904 i_blocks 45063 00000020:00200000:1.0:1789760721.681768:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760721.681779:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760721.681805:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23035904, count: 4096 00000080:00200000:1.0:1789760721.681812:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23035904, bytes: 4096 00000080:00200000:1.0:1789760721.681822:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760721.681829:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760721.681831:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23035904, 23040000) 0 0 00000080:00200000:1.0:1789760721.681835:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760721.681848:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5624, 5624] 00020000:00200000:1.0:1789760721.681854:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23035904 chunk: [23035904, 23040000) 23040000, 4096 00020000:00200000:1.0:1789760721.681858:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760721.681873:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5624, 5624] 00000008:00010000:1.0:1789760721.681887:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760721.681892:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760721.681929:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23035904, 23040000) append 0 append lockpos 0 00000080:00200000:1.0:1789760721.681940:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23035904=0x15f8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760721.681987:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760721.681990:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23040000 00000020:00010000:1.0:1789760721.681996:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760721.681999:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.682002:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.682004:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760721.682007:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760721.682014:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760721.682020:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760721.682036:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760721.682110:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 23035904<>23040000 00000080:00200000:3.0:1789760721.693245:0:38:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1789760721.693257:0:38:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760721.693279:0:38:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760721.719172:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5624, 5624] 00000080:00200000:1.0:1789760721.719184:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760721.719189:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760721.719200:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23040000 i_blocks 45063 00000020:00200000:1.0:1789760721.719204:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760721.719211:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760721.719237:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23040000, count: 4096 00000080:00200000:1.0:1789760721.719243:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23040000, bytes: 4096 00000080:00200000:1.0:1789760721.719250:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760721.719256:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760721.719258:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23040000, 23044096) 0 0 00000080:00200000:1.0:1789760721.719262:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760721.719274:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5625, 5625] 00020000:00200000:1.0:1789760721.719281:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23040000 chunk: [23040000, 23044096) 23044096, 4096 00020000:00200000:1.0:1789760721.719284:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760721.719298:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5625, 5625] 00000008:00010000:1.0:1789760721.719310:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760721.719315:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760721.719323:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23040000, 23044096) append 0 append lockpos 0 00000080:00200000:1.0:1789760721.719333:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23040000=0x15f9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760721.719378:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760721.719381:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23044096 00000020:00010000:1.0:1789760721.719387:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760721.719390:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.719393:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.719396:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760721.719399:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760721.719405:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760721.719410:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760721.719422:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:1.0:1789760721.719497:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95c00 - object 0x240000400:5 offset 23040000<>23044096 00000080:00200000:1.0:1789760721.752563:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5625, 5625] 00000080:00200000:1.0:1789760721.752590:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760721.752596:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760721.752608:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23044096 i_blocks 45063 00000020:00200000:1.0:1789760721.752612:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760721.752622:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760721.752644:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23044096, count: 4096 00000080:00200000:1.0:1789760721.752650:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23044096, bytes: 4096 00000080:00200000:1.0:1789760721.752660:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760721.752668:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760721.752670:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23044096, 23048192) 0 0 00000080:00200000:1.0:1789760721.752674:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760721.752687:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5626, 5626] 00020000:00200000:1.0:1789760721.752694:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23044096 chunk: [23044096, 23048192) 23048192, 4096 00020000:00200000:1.0:1789760721.752697:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760721.752713:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5626, 5626] 00000008:00010000:1.0:1789760721.752728:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760721.752733:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760721.752742:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23044096, 23048192) append 0 append lockpos 0 00000080:00200000:1.0:1789760721.752754:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23044096=0x15fa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760721.752801:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760721.752804:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23048192 00000020:00010000:1.0:1789760721.752810:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760721.752814:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.752816:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760721.752819:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760721.752822:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760721.752828:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760721.752833:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760721.753313:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42900 00000008:00100000:1.0:1789760721.753407:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 23044096<>23048192 00000100:00100000:1.0:1789760721.753458:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512579200:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760721.784813:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972945 gen 3 00000008:00010000:1.0:1789760721.784851:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42900 00000020:00200000:1.0:1789760721.784858:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760721.784865:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760721.784885:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42900 00000100:00100000:1.0:1789760721.784943:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512579200:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760721.786296:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760721.786315:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512579328:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760721.810730:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972946 gen 3 00000008:00010000:1.0:1789760721.810765:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760721.810772:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760721.810782:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760721.810805:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760721.810826:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512579328:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760721.811373:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760721.811388:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512579456:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760721.851798:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972947 gen 3 00000008:00010000:1.0:1789760721.851833:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760721.851840:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760721.851848:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760721.851870:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760721.851892:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512579456:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760721.854762:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760721.854796:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512579584:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760721.894783:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972948 gen 3 00000008:00010000:1.0:1789760721.894833:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760721.894839:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760721.894849:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760721.894869:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760721.894890:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512579584:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760721.895674:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760721.895689:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512579712:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760721.931640:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972949 gen 3 00000008:00010000:1.0:1789760721.931678:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760721.931685:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760721.931695:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760721.931716:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760721.931739:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512579712:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760721.933621:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760721.933640:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512579840:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760721.971184:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972950 gen 3 00000008:00010000:1.0:1789760721.971223:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760721.971229:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760721.971239:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760721.971267:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760721.971290:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512579840:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760721.971430:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5631, 5631] 00000080:00200000:3.0:1789760721.971442:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760721.971447:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760721.971459:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23068672 i_blocks 45063 00000020:00200000:3.0:1789760721.971462:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760721.971469:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760721.971495:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23068672, count: 4096 00000080:00200000:3.0:1789760721.971501:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23068672, bytes: 4096 00000080:00200000:3.0:1789760721.971508:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760721.971515:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760721.971517:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23068672, 23072768) 0 0 00000080:00200000:3.0:1789760721.971521:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760721.971534:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5632, 5632] 00020000:00200000:3.0:1789760721.971540:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23068672 chunk: [23068672, 23072768) 23072768, 4096 00020000:00200000:3.0:1789760721.971544:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760721.971558:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5632, 5632] 00000008:00010000:3.0:1789760721.971582:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760721.971587:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760721.971596:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23068672, 23072768) append 0 append lockpos 0 00000080:00200000:3.0:1789760721.971607:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23068672=0x1600000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760721.971652:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760721.971655:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23072768 00000020:00010000:3.0:1789760721.971661:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760721.971665:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.971668:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760721.971671:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760721.971674:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760721.971680:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760721.971685:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760721.971699:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760721.971781:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 23068672<>23072768 00000100:00100000:3.0:1789760721.971829:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760721.971843:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512579968:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760722.013770:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972951 gen 3 00000008:00010000:3.0:1789760722.013813:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:3.0:1789760722.013819:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760722.013827:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760722.013847:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:3.0:1789760722.013867:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512579968:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.015748:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760722.015767:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512580096:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760722.057601:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760722.057612:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.057621:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.057641:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760722.057664:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512580096:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.058719:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760722.058739:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512580224:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.102264:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972952 gen 3 00000100:00100000:1.0:1789760722.102295:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00514ad80 x1876699512580224/t4294972953(4294972953) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760738 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760722.102333:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760722.102339:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.102348:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.102372:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760722.102404:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512580224:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.104127:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760722.104148:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512580352:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.142992:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972954 gen 3 00000008:00010000:1.0:1789760722.143038:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760722.143044:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.143053:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.143072:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760722.143092:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512580352:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.143988:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760722.144004:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512580480:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.179867:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972955 gen 3 00000008:00010000:1.0:1789760722.179919:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760722.179924:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.179932:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.179946:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760722.179963:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512580480:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.180491:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512580608:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.217096:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972956 gen 3 00000008:00010000:1.0:1789760722.217130:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760722.217137:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.217145:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.217166:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760722.217187:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512580608:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.218468:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512580736:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.259088:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972957 gen 3 00000008:00010000:1.0:1789760722.259125:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760722.259131:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.259140:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.259160:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760722.259192:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512580736:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.260046:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760722.260064:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512580864:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.289707:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972958 gen 3 00000008:00010000:1.0:1789760722.289745:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760722.289752:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.289761:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.289934:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760722.289957:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512580864:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.290471:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512580992:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760722.334036:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760722.334050:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.334060:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.334085:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760722.334112:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512580992:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.334634:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512581120:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.373329:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972959 gen 3 00000100:00100000:1.0:1789760722.373360:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00514b480 x1876699512581120/t4294972960(4294972960) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760738 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760722.373399:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760722.373405:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.373414:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.373489:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760722.373508:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512581120:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.374538:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512581248:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.429254:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972961 gen 3 00000008:00010000:1.0:1789760722.429299:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760722.429306:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.429313:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.429334:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760722.429355:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512581248:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760722.429578:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5642, 5642] 00000080:00200000:3.0:1789760722.429590:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760722.429595:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760722.429606:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23113728 i_blocks 47111 00000020:00200000:3.0:1789760722.429609:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760722.429617:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760722.429640:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23113728, count: 4096 00000080:00200000:3.0:1789760722.429646:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23113728, bytes: 4096 00000080:00200000:3.0:1789760722.429653:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760722.429660:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760722.429662:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23113728, 23117824) 0 0 00000080:00200000:3.0:1789760722.429665:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760722.429676:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5643, 5643] 00020000:00200000:3.0:1789760722.429682:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23113728 chunk: [23113728, 23117824) 23117824, 4096 00020000:00200000:3.0:1789760722.429686:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760722.429699:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5643, 5643] 00000008:00010000:3.0:1789760722.429712:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760722.429717:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760722.429724:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23113728, 23117824) append 0 append lockpos 0 00000080:00200000:3.0:1789760722.429734:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23113728=0x160b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760722.429773:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760722.429776:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23117824 00000020:00010000:3.0:1789760722.429782:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760722.429785:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760722.429787:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760722.429790:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760722.429792:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760722.429797:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760722.429801:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760722.429813:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760722.429886:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 23113728<>23117824 00000100:00100000:3.0:1789760722.429962:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760722.429975:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512581376:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760722.523957:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972962 gen 3 00000008:00010000:3.0:1789760722.524002:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:3.0:1789760722.524008:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760722.524016:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760722.524030:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:3.0:1789760722.524053:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512581376:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760722.524167:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5643, 5643] 00000080:00200000:2.0:1789760722.524179:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760722.524184:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760722.524196:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23117824 i_blocks 47111 00000020:00200000:2.0:1789760722.524199:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760722.524207:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760722.524233:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23117824, count: 4096 00000080:00200000:2.0:1789760722.524247:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23117824, bytes: 4096 00000080:00200000:2.0:1789760722.524256:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760722.524263:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760722.524265:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23117824, 23121920) 0 0 00000080:00200000:2.0:1789760722.524268:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760722.524281:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5644, 5644] 00020000:00200000:2.0:1789760722.524287:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23117824 chunk: [23117824, 23121920) 23121920, 4096 00020000:00200000:2.0:1789760722.524291:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760722.524305:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5644, 5644] 00000008:00010000:2.0:1789760722.524317:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760722.524322:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760722.524330:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23117824, 23121920) append 0 append lockpos 0 00000080:00200000:2.0:1789760722.524340:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23117824=0x160c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760722.524385:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760722.524387:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23121920 00000020:00010000:2.0:1789760722.524395:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760722.524398:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760722.524400:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760722.524403:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760722.524406:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760722.524413:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760722.524419:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760722.524434:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000008:00100000:2.0:1789760722.524514:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 23117824<>23121920 00000100:00100000:2.0:1789760722.524552:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760722.524574:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512581504:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760722.564261:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972963 gen 3 00000008:00010000:3.0:1789760722.564299:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760722.564306:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760722.564315:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760722.564331:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760722.564354:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512581504:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760722.564433:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5644, 5644] 00000080:00200000:2.0:1789760722.564446:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760722.564451:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760722.564462:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23121920 i_blocks 47111 00000020:00200000:2.0:1789760722.564464:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760722.564472:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760722.564498:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23121920, count: 4096 00000080:00200000:2.0:1789760722.564504:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23121920, bytes: 4096 00000080:00200000:2.0:1789760722.564512:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760722.564518:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760722.564520:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23121920, 23126016) 0 0 00000080:00200000:2.0:1789760722.564524:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760722.564538:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5645, 5645] 00020000:00200000:2.0:1789760722.564543:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23121920 chunk: [23121920, 23126016) 23126016, 4096 00020000:00200000:2.0:1789760722.564547:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760722.564562:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5645, 5645] 00000008:00010000:2.0:1789760722.564585:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760722.564590:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760722.564598:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23121920, 23126016) append 0 append lockpos 0 00000080:00200000:2.0:1789760722.564609:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23121920=0x160d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760722.564652:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760722.564654:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23126016 00000020:00010000:2.0:1789760722.564661:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760722.564665:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760722.564668:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760722.564671:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760722.564674:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760722.564680:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760722.564685:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760722.564699:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760722.564773:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 23121920<>23126016 00000100:00100000:3.0:1789760722.564974:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512581632:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760722.610170:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972964 gen 3 00000008:00010000:3.0:1789760722.610207:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760722.610213:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760722.610221:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760722.610246:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760722.610269:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512581632:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760722.610403:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5645, 5645] 00000080:00200000:2.0:1789760722.610415:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760722.610421:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760722.610434:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23126016 i_blocks 47111 00000020:00200000:2.0:1789760722.610437:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760722.610445:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760722.610472:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23126016, count: 4096 00000080:00200000:2.0:1789760722.610479:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23126016, bytes: 4096 00000080:00200000:2.0:1789760722.610486:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760722.610494:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760722.610496:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23126016, 23130112) 0 0 00000080:00200000:2.0:1789760722.610499:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760722.610512:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5646, 5646] 00020000:00200000:2.0:1789760722.610519:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23126016 chunk: [23126016, 23130112) 23130112, 4096 00020000:00200000:2.0:1789760722.610523:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760722.610539:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5646, 5646] 00000008:00010000:2.0:1789760722.610553:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760722.610559:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760722.610576:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23126016, 23130112) append 0 append lockpos 0 00000080:00200000:2.0:1789760722.610588:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23126016=0x160e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760722.610631:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760722.610633:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23130112 00000020:00010000:2.0:1789760722.610639:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760722.610642:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760722.610645:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760722.610648:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760722.610650:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760722.610657:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760722.610662:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760722.610676:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760722.610751:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 23126016<>23130112 00000100:00100000:3.0:1789760722.610830:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760722.610845:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512581760:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760722.648366:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972965 gen 3 00000008:00010000:3.0:1789760722.648400:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760722.648408:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760722.648416:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760722.648432:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760722.648455:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512581760:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760722.648597:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5646, 5646] 00000080:00200000:2.0:1789760722.648611:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760722.648616:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760722.648628:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23130112 i_blocks 47111 00000020:00200000:2.0:1789760722.648632:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760722.648640:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760722.648667:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23130112, count: 4096 00000080:00200000:2.0:1789760722.648673:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23130112, bytes: 4096 00000080:00200000:2.0:1789760722.648679:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760722.648686:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760722.648688:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23130112, 23134208) 0 0 00000080:00200000:2.0:1789760722.648691:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760722.648705:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5647, 5647] 00020000:00200000:2.0:1789760722.648712:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23130112 chunk: [23130112, 23134208) 23134208, 4096 00020000:00200000:2.0:1789760722.648716:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760722.648731:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5647, 5647] 00000008:00010000:2.0:1789760722.648746:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760722.648750:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760722.648760:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23130112, 23134208) append 0 append lockpos 0 00000080:00200000:2.0:1789760722.648770:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23130112=0x160f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760722.648821:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760722.648824:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23134208 00000020:00010000:2.0:1789760722.648831:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760722.648834:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760722.648836:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760722.648839:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760722.648842:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760722.648849:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760722.648854:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760722.648867:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000008:00100000:2.0:1789760722.648982:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 23130112<>23134208 00000100:00100000:3.0:1789760722.649040:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512581888:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760722.688117:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760722.688129:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760722.688137:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760722.688226:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760722.688251:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512581888:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.688937:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760722.688953:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512582016:192.168.203.156@tcp:4:dd.0 00010000:00010000:3.0:1789760722.719237:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760722.735120:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972967 gen 3 00000008:00010000:1.0:1789760722.735163:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760722.735168:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.735174:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.735237:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760722.735259:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512582016:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.735868:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512582144:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.779776:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972968 gen 3 00000008:00010000:1.0:1789760722.779811:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760722.779817:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.779826:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.779847:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760722.779877:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512582144:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.780701:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512582272:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.837720:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972969 gen 3 00000008:00010000:1.0:1789760722.837758:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760722.837765:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.837775:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.837802:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760722.837825:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512582272:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.838625:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512582400:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.874490:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972970 gen 3 00000008:00010000:1.0:1789760722.874529:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760722.874535:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.874543:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.874619:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760722.874642:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512582400:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.876471:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512582528:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.928190:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972971 gen 3 00000008:00010000:1.0:1789760722.928225:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760722.928231:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.928240:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.928262:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760722.928282:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512582528:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.929880:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512582656:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760722.968029:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760722.968040:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760722.968048:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760722.968068:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760722.968090:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512582656:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760722.968628:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760722.968639:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512582784:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760723.033553:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972973 gen 3 00000008:00010000:1.0:1789760723.033631:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760723.033639:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760723.033650:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760723.033676:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760723.033706:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512582784:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760723.034831:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512582912:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760723.109610:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972974 gen 3 00000008:00010000:1.0:1789760723.109659:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760723.109666:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760723.109677:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760723.109701:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760723.109730:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512582912:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760723.110896:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512583040:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760723.174199:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972975 gen 3 00000008:00010000:1.0:1789760723.174262:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760723.174270:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760723.174282:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760723.174304:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760723.174342:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512583040:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760723.176212:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512583168:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760723.256125:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972976 gen 3 00000008:00010000:1.0:1789760723.256187:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760723.256196:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760723.256208:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760723.256235:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760723.256269:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512583168:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760723.261217:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5657, 5657] 00000080:00200000:2.0:1789760723.261244:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760723.261252:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760723.261270:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23175168 i_blocks 47111 00000020:00200000:2.0:1789760723.261276:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760723.261292:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760723.261341:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23175168, count: 4096 00000080:00200000:2.0:1789760723.261350:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23175168, bytes: 4096 00000080:00200000:2.0:1789760723.261366:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760723.261375:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760723.261378:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23175168, 23179264) 0 0 00000080:00200000:2.0:1789760723.261382:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760723.261400:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5658, 5658] 00020000:00200000:2.0:1789760723.261409:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23175168 chunk: [23175168, 23179264) 23179264, 4096 00020000:00200000:2.0:1789760723.261413:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760723.261436:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5658, 5658] 00000008:00010000:2.0:1789760723.261467:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760723.261474:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760723.261488:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23175168, 23179264) append 0 append lockpos 0 00000080:00200000:2.0:1789760723.261504:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23175168=0x161a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760723.261583:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760723.261586:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23179264 00000020:00010000:2.0:1789760723.261595:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760723.261598:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760723.261601:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760723.261605:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760723.261608:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760723.261616:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760723.261622:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760723.261644:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760723.261773:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 23175168<>23179264 00000100:00100000:3.0:1789760723.261889:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760723.261972:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512583296:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760723.358586:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972977 gen 3 00000008:00010000:3.0:1789760723.358695:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760723.358704:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760723.358715:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760723.358739:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760723.358771:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512583296:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760723.359208:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5658, 5658] 00000080:00200000:2.0:1789760723.359227:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760723.359232:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760723.359247:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23179264 i_blocks 47111 00000020:00200000:2.0:1789760723.359252:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760723.359262:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760723.359307:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23179264, count: 4096 00000080:00200000:2.0:1789760723.359315:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23179264, bytes: 4096 00000080:00200000:2.0:1789760723.359326:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760723.359335:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760723.359337:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23179264, 23183360) 0 0 00000080:00200000:2.0:1789760723.359342:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760723.359359:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5659, 5659] 00020000:00200000:2.0:1789760723.359368:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23179264 chunk: [23179264, 23183360) 23183360, 4096 00020000:00200000:2.0:1789760723.359373:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760723.359394:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5659, 5659] 00000008:00010000:2.0:1789760723.359414:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760723.359421:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760723.359433:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23179264, 23183360) append 0 append lockpos 0 00000080:00200000:2.0:1789760723.359447:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23179264=0x161b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760723.359500:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760723.359502:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23183360 00000020:00010000:2.0:1789760723.359509:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760723.359511:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760723.359513:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760723.359516:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760723.359518:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760723.359524:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760723.359529:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760723.359542:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760723.359775:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 23179264<>23183360 00000100:00100000:3.0:1789760723.360336:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512583424:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760723.418401:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972978 gen 3 00000008:00010000:3.0:1789760723.418454:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760723.418463:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760723.418475:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760723.418498:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760723.418530:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512583424:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760723.418767:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5659, 5659] 00000080:00200000:2.0:1789760723.418784:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760723.418791:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760723.418806:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23183360 i_blocks 47111 00000020:00200000:2.0:1789760723.418810:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760723.418820:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760723.418854:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23183360, count: 4096 00000080:00200000:2.0:1789760723.418861:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23183360, bytes: 4096 00000080:00200000:2.0:1789760723.418875:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760723.418884:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760723.418886:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23183360, 23187456) 0 0 00000080:00200000:2.0:1789760723.418891:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760723.419222:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5660, 5660] 00020000:00200000:2.0:1789760723.419233:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23183360 chunk: [23183360, 23187456) 23187456, 4096 00020000:00200000:2.0:1789760723.419237:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760723.419260:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5660, 5660] 00000008:00010000:2.0:1789760723.419277:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760723.419284:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760723.419297:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23183360, 23187456) append 0 append lockpos 0 00000080:00200000:2.0:1789760723.419311:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23183360=0x161c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760723.419372:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760723.419375:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23187456 00000020:00010000:2.0:1789760723.419438:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760723.419443:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760723.419445:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760723.419448:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760723.419451:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760723.419459:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760723.419466:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760723.419486:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760723.419667:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 23183360<>23187456 00000100:00100000:3.0:1789760723.419779:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760723.419799:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512583552:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760723.482679:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972979 gen 3 00000008:00010000:3.0:1789760723.482733:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760723.482741:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760723.482753:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760723.482774:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760723.482807:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512583552:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760723.483091:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5660, 5660] 00000080:00200000:2.0:1789760723.483106:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760723.483113:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760723.483128:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23187456 i_blocks 47111 00000020:00200000:2.0:1789760723.483132:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760723.483142:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760723.483177:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23187456, count: 4096 00000080:00200000:2.0:1789760723.483184:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23187456, bytes: 4096 00000080:00200000:2.0:1789760723.483194:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760723.483203:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760723.483205:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23187456, 23191552) 0 0 00000080:00200000:2.0:1789760723.483209:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760723.483227:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5661, 5661] 00020000:00200000:2.0:1789760723.483236:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23187456 chunk: [23187456, 23191552) 23191552, 4096 00020000:00200000:2.0:1789760723.483240:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760723.483261:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5661, 5661] 00000008:00010000:2.0:1789760723.483279:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760723.483298:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760723.483314:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23187456, 23191552) append 0 append lockpos 0 00000080:00200000:2.0:1789760723.483328:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23187456=0x161d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760723.483392:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760723.483394:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23191552 00000020:00010000:2.0:1789760723.483403:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760723.483407:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760723.483409:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760723.483413:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760723.483415:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760723.483422:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760723.483428:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760723.483444:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760723.483716:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 23187456<>23191552 00000100:00100000:3.0:1789760723.483779:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512583680:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760723.565426:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972980 gen 3 00000008:00010000:3.0:1789760723.565480:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760723.565489:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760723.565500:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760723.565534:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760723.565579:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512583680:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760723.566440:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512583808:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760723.643026:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760723.643041:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760723.643054:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760723.643078:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760723.643109:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512583808:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760723.644364:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512583936:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760723.705328:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972982 gen 3 00000008:00010000:1.0:1789760723.705393:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760723.705402:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760723.705414:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760723.705439:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760723.705467:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512583936:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760723.706503:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512584064:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760723.742410:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512584320:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760723.742718:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1789760723.742726:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e6cf7267-9b19-462a-a263-14115df15dab:2361:1876699512584192:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760723.742764:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2361:1876699512584448:192.168.203.156@tcp:400:kworker.0 00000100:00100000:2.0:1789760723.748336:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2361:1876699512584448:192.168.203.156@tcp:400:kworker.0 00000100:00100000:2.0:1789760723.748355:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512584320:192.168.203.156@tcp:400:kworker.0 00000100:00100000:2.0:1789760723.762865:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e6cf7267-9b19-462a-a263-14115df15dab:2361:1876699512584192:192.168.203.156@tcp:400:kworker.0 00000100:00100000:1.0:1789760723.776414:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972983 gen 3 00000008:00010000:1.0:1789760723.776633:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760723.776642:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760723.776654:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760723.776737:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760723.776772:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512584064:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760723.778036:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760723.778061:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512584576:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760723.818433:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972984 gen 3 00000008:00010000:1.0:1789760723.818485:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760723.818494:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760723.818507:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760723.818530:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760723.818568:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512584576:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760723.821783:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512584704:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760723.878221:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972985 gen 3 00000008:00010000:1.0:1789760723.878274:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760723.878283:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760723.878297:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760723.878320:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760723.878350:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512584704:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760723.879659:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760723.879680:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512584832:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760723.923579:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972986 gen 3 00000008:00010000:1.0:1789760723.923638:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760723.924036:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760723.924049:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760723.924074:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760723.924109:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512584832:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760723.925101:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cce40 00000008:00100000:1.0:1789760723.926045:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 23216128<>23220224 00000100:00100000:1.0:1789760723.926893:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512584960:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760723.971477:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972987 gen 3 00000008:00010000:1.0:1789760723.971529:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760723.971539:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760723.971552:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760723.971585:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760723.971612:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512584960:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760723.972564:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512585088:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760724.028728:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972988 gen 3 00000008:00010000:1.0:1789760724.028781:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760724.028790:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760724.028802:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760724.028825:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760724.028854:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512585088:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760724.031077:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760724.031110:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512585216:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760724.108873:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972989 gen 3 00000008:00010000:1.0:1789760724.108979:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760724.108987:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760724.109000:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760724.109024:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760724.109056:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512585216:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760724.109881:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc180 00000008:00100000:1.0:1789760724.110021:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 23228416<>23232512 00000100:00100000:1.0:1789760724.110185:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512585344:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760724.169626:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760724.169643:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760724.169657:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760724.169680:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760724.169710:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512585344:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760724.170648:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512585472:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760724.210773:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972990 gen 3 00000100:00100000:1.0:1789760724.210820:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00514a300 x1876699512585472/t4294972991(4294972991) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760740 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760724.210874:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760724.210882:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760724.210893:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760724.210955:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760724.210981:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512585472:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760724.211875:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760724.211895:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512585600:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760724.270201:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972992 gen 3 00000008:00010000:1.0:1789760724.270281:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760724.270290:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760724.270301:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760724.270324:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760724.270355:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512585600:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760724.271236:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512585728:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760724.328220:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972993 gen 3 00000008:00010000:1.0:1789760724.328280:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760724.328288:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760724.328300:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760724.328321:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760724.328350:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512585728:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760724.329299:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512585856:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760724.387617:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760724.387634:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760724.387646:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760724.387671:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760724.387703:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512585856:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760724.387800:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5675, 5675] 00000080:00200000:3.0:1789760724.387816:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760724.387822:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760724.387837:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23248896 i_blocks 47111 00000020:00200000:3.0:1789760724.387841:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760724.387851:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760724.387886:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23248896, count: 4096 00000080:00200000:3.0:1789760724.387893:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23248896, bytes: 4096 00000080:00200000:3.0:1789760724.387934:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760724.387953:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760724.387955:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23248896, 23252992) 0 0 00000080:00200000:3.0:1789760724.387960:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760724.387978:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5676, 5676] 00020000:00200000:3.0:1789760724.387986:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23248896 chunk: [23248896, 23252992) 23252992, 4096 00020000:00200000:3.0:1789760724.387992:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760724.388012:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5676, 5676] 00000008:00010000:3.0:1789760724.388031:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760724.388037:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760724.388050:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23248896, 23252992) append 0 append lockpos 0 00000080:00200000:3.0:1789760724.388065:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23248896=0x162c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760724.388125:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760724.388128:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23252992 00000020:00010000:3.0:1789760724.388137:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760724.388140:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.388143:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.388147:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760724.388150:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760724.388158:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760724.388164:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760724.388180:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760724.388404:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 23248896<>23252992 00000100:00100000:2.0:1789760724.388980:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760724.389007:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512585984:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760724.429849:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972995 gen 3 00000008:00010000:2.0:1789760724.430040:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760724.430050:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760724.430061:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760724.430123:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:2.0:1789760724.430154:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512585984:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760724.430287:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5676, 5676] 00000080:00200000:3.0:1789760724.430305:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760724.430310:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760724.430325:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23252992 i_blocks 47111 00000020:00200000:3.0:1789760724.430329:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760724.430339:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760724.430374:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23252992, count: 4096 00000080:00200000:3.0:1789760724.430382:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23252992, bytes: 4096 00000080:00200000:3.0:1789760724.430392:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760724.430400:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760724.430402:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23252992, 23257088) 0 0 00000080:00200000:3.0:1789760724.430407:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760724.430596:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5677, 5677] 00020000:00200000:3.0:1789760724.430604:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23252992 chunk: [23252992, 23257088) 23257088, 4096 00020000:00200000:3.0:1789760724.430609:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760724.430628:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5677, 5677] 00000008:00010000:3.0:1789760724.430644:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760724.430649:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760724.430660:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23252992, 23257088) append 0 append lockpos 0 00000080:00200000:3.0:1789760724.430672:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23252992=0x162d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760724.430731:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760724.430734:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23257088 00000020:00010000:3.0:1789760724.430742:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760724.430746:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.430749:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.430753:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760724.430757:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760724.430765:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760724.430772:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760724.431433:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760724.431546:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107afb80 - object 0x240000400:5 offset 23252992<>23257088 00000100:00100000:2.0:1789760724.431626:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512586112:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760724.475943:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972996 gen 3 00000008:00010000:2.0:1789760724.476006:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760724.476015:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760724.476028:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760724.476048:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760724.476078:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512586112:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760724.476186:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5677, 5677] 00000080:00200000:3.0:1789760724.476205:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760724.476211:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760724.476237:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23257088 i_blocks 47111 00000020:00200000:3.0:1789760724.476241:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760724.476252:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760724.476288:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23257088, count: 4096 00000080:00200000:3.0:1789760724.476295:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23257088, bytes: 4096 00000080:00200000:3.0:1789760724.476305:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760724.476313:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760724.476316:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23257088, 23261184) 0 0 00000080:00200000:3.0:1789760724.476320:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760724.476336:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5678, 5678] 00020000:00200000:3.0:1789760724.476346:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23257088 chunk: [23257088, 23261184) 23261184, 4096 00020000:00200000:3.0:1789760724.476350:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760724.476371:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5678, 5678] 00000008:00010000:3.0:1789760724.476390:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760724.476395:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760724.476408:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23257088, 23261184) append 0 append lockpos 0 00000080:00200000:3.0:1789760724.476422:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23257088=0x162e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760724.476481:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760724.476484:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23261184 00000020:00010000:3.0:1789760724.476493:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760724.476496:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.476499:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.476502:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760724.476505:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760724.476513:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760724.476519:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760724.476536:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760724.476664:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad500 - object 0x240000400:5 offset 23257088<>23261184 00000100:00100000:2.0:1789760724.476833:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512586240:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760724.523355:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972997 gen 3 00000008:00010000:2.0:1789760724.523403:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760724.523411:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760724.523423:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760724.523444:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760724.523483:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512586240:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760724.524123:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5678, 5678] 00000080:00200000:3.0:1789760724.524140:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760724.524147:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760724.524161:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23261184 i_blocks 47111 00000020:00200000:3.0:1789760724.524165:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760724.524176:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760724.524208:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23261184, count: 4096 00000080:00200000:3.0:1789760724.524216:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23261184, bytes: 4096 00000080:00200000:3.0:1789760724.524227:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760724.524236:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760724.524238:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23261184, 23265280) 0 0 00000080:00200000:3.0:1789760724.524242:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760724.524262:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5679, 5679] 00020000:00200000:3.0:1789760724.524275:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23261184 chunk: [23261184, 23265280) 23265280, 4096 00020000:00200000:3.0:1789760724.524281:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760724.524304:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5679, 5679] 00000008:00010000:3.0:1789760724.524325:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760724.524331:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760724.524343:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23261184, 23265280) append 0 append lockpos 0 00000080:00200000:3.0:1789760724.524357:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23261184=0x162f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760724.524423:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760724.524426:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23265280 00000020:00010000:3.0:1789760724.524435:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760724.524438:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.524441:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.524445:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760724.524447:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760724.524455:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760724.524462:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760724.524491:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760724.524607:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 23261184<>23265280 00000100:00100000:2.0:1789760724.524735:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512586368:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760724.564242:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294972998 gen 3 00000008:00010000:2.0:1789760724.564296:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760724.564305:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760724.564319:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760724.564335:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760724.564365:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512586368:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760724.564820:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5679, 5679] 00000080:00200000:3.0:1789760724.564839:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760724.564845:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760724.564861:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23265280 i_blocks 47111 00000020:00200000:3.0:1789760724.564866:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760724.564876:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760724.564939:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23265280, count: 4096 00000080:00200000:3.0:1789760724.564957:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23265280, bytes: 4096 00000080:00200000:3.0:1789760724.564968:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760724.564977:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760724.564979:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23265280, 23269376) 0 0 00000080:00200000:3.0:1789760724.564984:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760724.565001:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5680, 5680] 00020000:00200000:3.0:1789760724.565009:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23265280 chunk: [23265280, 23269376) 23269376, 4096 00020000:00200000:3.0:1789760724.565013:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760724.565032:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5680, 5680] 00000008:00010000:3.0:1789760724.565049:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760724.565055:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760724.565066:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23265280, 23269376) append 0 append lockpos 0 00000080:00200000:3.0:1789760724.565080:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23265280=0x1630000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760724.565144:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760724.565147:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23269376 00000020:00010000:3.0:1789760724.565156:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760724.565159:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.565163:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.565167:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760724.565170:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760724.565178:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760724.565184:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760724.565203:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760724.565320:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac700 - object 0x240000400:5 offset 23265280<>23269376 00000100:00100000:2.0:1789760724.565538:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512586496:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760724.611498:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760724.611514:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760724.611527:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760724.611547:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760724.611587:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512586496:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760724.611934:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5680, 5680] 00000080:00200000:3.0:1789760724.611959:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760724.611966:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760724.611981:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23269376 i_blocks 47111 00000020:00200000:3.0:1789760724.611986:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760724.611996:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760724.612033:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23269376, count: 4096 00000080:00200000:3.0:1789760724.612041:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23269376, bytes: 4096 00000080:00200000:3.0:1789760724.612052:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760724.612060:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760724.612062:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23269376, 23273472) 0 0 00000080:00200000:3.0:1789760724.612067:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760724.612083:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5681, 5681] 00020000:00200000:3.0:1789760724.612092:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23269376 chunk: [23269376, 23273472) 23273472, 4096 00020000:00200000:3.0:1789760724.612097:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760724.612114:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5681, 5681] 00000008:00010000:3.0:1789760724.612131:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760724.612136:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760724.612147:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23269376, 23273472) append 0 append lockpos 0 00000080:00200000:3.0:1789760724.612161:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23269376=0x1631000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760724.612232:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760724.612235:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23273472 00000020:00010000:3.0:1789760724.612244:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760724.612248:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.612251:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.612254:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760724.612257:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760724.612265:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760724.612272:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760724.612290:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760724.612397:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af480 - object 0x240000400:5 offset 23269376<>23273472 00000100:00100000:2.0:1789760724.612573:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512586624:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760724.661353:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973000 gen 3 00000008:00010000:2.0:1789760724.661421:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760724.661429:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760724.661440:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760724.661460:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:2.0:1789760724.661490:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512586624:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760724.661652:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5681, 5681] 00000080:00200000:3.0:1789760724.661668:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760724.661674:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760724.661689:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23273472 i_blocks 47111 00000020:00200000:3.0:1789760724.661692:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760724.661703:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760724.661739:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23273472, count: 4096 00000080:00200000:3.0:1789760724.661746:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23273472, bytes: 4096 00000080:00200000:3.0:1789760724.661756:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760724.661764:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760724.661766:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23273472, 23277568) 0 0 00000080:00200000:3.0:1789760724.661770:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760724.661787:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5682, 5682] 00020000:00200000:3.0:1789760724.661797:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23273472 chunk: [23273472, 23277568) 23277568, 4096 00020000:00200000:3.0:1789760724.661801:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760724.661820:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5682, 5682] 00000008:00010000:3.0:1789760724.661837:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760724.661844:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760724.661856:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23273472, 23277568) append 0 append lockpos 0 00000080:00200000:3.0:1789760724.661869:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23273472=0x1632000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760724.661975:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760724.661978:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23277568 00000020:00010000:3.0:1789760724.661986:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760724.661990:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.661992:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.661996:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760724.661998:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760724.662005:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760724.662011:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760724.662028:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760724.662131:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae300 - object 0x240000400:5 offset 23273472<>23277568 00000100:00100000:2.0:1789760724.662295:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512586752:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760724.705535:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973001 gen 3 00000008:00010000:2.0:1789760724.705599:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760724.705607:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760724.705619:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760724.705641:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760724.705672:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512586752:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760724.705779:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5682, 5682] 00000080:00200000:3.0:1789760724.705799:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760724.705805:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760724.705819:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23277568 i_blocks 47111 00000020:00200000:3.0:1789760724.705823:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760724.705831:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760724.705862:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23277568, count: 4096 00000080:00200000:3.0:1789760724.705868:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23277568, bytes: 4096 00000080:00200000:3.0:1789760724.705876:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760724.705885:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760724.705888:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23277568, 23281664) 0 0 00000080:00200000:3.0:1789760724.705892:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760724.705940:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5683, 5683] 00020000:00200000:3.0:1789760724.705958:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23277568 chunk: [23277568, 23281664) 23281664, 4096 00020000:00200000:3.0:1789760724.705962:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760724.705981:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5683, 5683] 00000008:00010000:3.0:1789760724.706000:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760724.706005:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760724.706018:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23277568, 23281664) append 0 append lockpos 0 00000080:00200000:3.0:1789760724.706031:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23277568=0x1633000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760724.706087:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760724.706090:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23281664 00000020:00010000:3.0:1789760724.706099:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760724.706102:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.706105:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.706109:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760724.706111:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760724.706118:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760724.706125:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760724.706141:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760724.706240:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac000 - object 0x240000400:5 offset 23277568<>23281664 00000100:00100000:2.0:1789760724.706387:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512586880:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760724.743758:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973002 gen 3 00000008:00010000:2.0:1789760724.743811:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760724.743820:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760724.743832:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760724.743850:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760724.743882:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512586880:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760724.744310:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5683, 5683] 00000080:00200000:3.0:1789760724.744328:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760724.744335:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760724.744352:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23281664 i_blocks 47111 00000020:00200000:3.0:1789760724.744356:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760724.744367:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760724.744401:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23281664, count: 4096 00000080:00200000:3.0:1789760724.744409:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23281664, bytes: 4096 00000080:00200000:3.0:1789760724.744419:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760724.744428:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760724.744430:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23281664, 23285760) 0 0 00000080:00200000:3.0:1789760724.744435:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760724.744451:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5684, 5684] 00020000:00200000:3.0:1789760724.744460:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23281664 chunk: [23281664, 23285760) 23285760, 4096 00020000:00200000:3.0:1789760724.744464:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760724.744484:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5684, 5684] 00000008:00010000:3.0:1789760724.744503:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760724.744508:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760724.744520:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23281664, 23285760) append 0 append lockpos 0 00000080:00200000:3.0:1789760724.744535:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23281664=0x1634000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760724.744608:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760724.744611:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23285760 00000020:00010000:3.0:1789760724.744620:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760724.744623:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.744626:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.744630:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760724.744633:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760724.744641:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760724.744649:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760724.744666:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760724.744777:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac380 - object 0x240000400:5 offset 23281664<>23285760 00000100:00100000:2.0:1789760724.744843:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512587008:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760724.811015:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760724.811033:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760724.811046:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760724.811113:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760724.811142:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512587008:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760724.811676:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5684, 5684] 00000080:00200000:3.0:1789760724.811693:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760724.811700:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760724.811716:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23285760 i_blocks 47111 00000020:00200000:3.0:1789760724.811720:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760724.811734:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760724.811836:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23285760, count: 4096 00000080:00200000:3.0:1789760724.811844:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23285760, bytes: 4096 00000080:00200000:3.0:1789760724.811856:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760724.811865:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760724.811867:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23285760, 23289856) 0 0 00000080:00200000:3.0:1789760724.811872:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760724.811889:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5685, 5685] 00020000:00200000:3.0:1789760724.811898:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23285760 chunk: [23285760, 23289856) 23289856, 4096 00020000:00200000:3.0:1789760724.811946:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760724.811967:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5685, 5685] 00000008:00010000:3.0:1789760724.811986:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760724.811992:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760724.812004:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23285760, 23289856) append 0 append lockpos 0 00000080:00200000:3.0:1789760724.812018:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23285760=0x1635000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760724.812077:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760724.812080:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23289856 00000020:00010000:3.0:1789760724.812087:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760724.812090:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.812093:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.812096:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760724.812099:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760724.812106:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760724.812112:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760724.812132:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760724.812232:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aea00 - object 0x240000400:5 offset 23285760<>23289856 00000100:00100000:2.0:1789760724.812308:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512587136:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760724.846452:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973003 gen 3 00000100:00100000:2.0:1789760724.846494:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc0107aea00 x1876699512587136/t4294973004(4294973004) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760740 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760724.846544:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760724.846561:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760724.846572:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760724.846667:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760724.846692:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512587136:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760724.846875:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5685, 5685] 00000080:00200000:3.0:1789760724.846894:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760724.846923:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760724.846939:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23289856 i_blocks 47111 00000020:00200000:3.0:1789760724.846943:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760724.846963:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760724.846999:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23289856, count: 4096 00000080:00200000:3.0:1789760724.847006:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23289856, bytes: 4096 00000080:00200000:3.0:1789760724.847021:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760724.847030:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760724.847032:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23289856, 23293952) 0 0 00000080:00200000:3.0:1789760724.847038:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760724.847056:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5686, 5686] 00020000:00200000:3.0:1789760724.847065:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23289856 chunk: [23289856, 23293952) 23293952, 4096 00020000:00200000:3.0:1789760724.847070:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760724.847090:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5686, 5686] 00000008:00010000:3.0:1789760724.847107:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760724.847113:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760724.847125:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23289856, 23293952) append 0 append lockpos 0 00000080:00200000:3.0:1789760724.847138:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23289856=0x1636000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760724.847195:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760724.847198:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23293952 00000020:00010000:3.0:1789760724.847206:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760724.847210:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.847213:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.847217:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760724.847219:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760724.847226:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760724.847232:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760724.847252:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760724.848365:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af100 - object 0x240000400:5 offset 23289856<>23293952 00000100:00100000:2.0:1789760724.848784:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512587264:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760724.917980:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973005 gen 3 00000008:00010000:2.0:1789760724.918055:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760724.918064:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760724.918076:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760724.918105:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760724.918138:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512587264:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760724.918257:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5686, 5686] 00000080:00200000:3.0:1789760724.918272:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760724.918278:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760724.918292:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23293952 i_blocks 47111 00000020:00200000:3.0:1789760724.918296:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760724.918305:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760724.918340:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23293952, count: 4096 00000080:00200000:3.0:1789760724.918347:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23293952, bytes: 4096 00000080:00200000:3.0:1789760724.918361:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760724.918371:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760724.918373:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23293952, 23298048) 0 0 00000080:00200000:3.0:1789760724.918378:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760724.918394:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5687, 5687] 00020000:00200000:3.0:1789760724.918403:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23293952 chunk: [23293952, 23298048) 23298048, 4096 00020000:00200000:3.0:1789760724.918407:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760724.918427:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5687, 5687] 00000008:00010000:3.0:1789760724.918446:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760724.918451:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760724.918463:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23293952, 23298048) append 0 append lockpos 0 00000080:00200000:3.0:1789760724.918478:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23293952=0x1637000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760724.918539:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760724.918542:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23298048 00000020:00010000:3.0:1789760724.918559:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760724.918563:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.918566:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760724.918569:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760724.918572:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760724.918580:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760724.918587:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760724.918604:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760724.918710:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad180 - object 0x240000400:5 offset 23293952<>23298048 00000100:00100000:2.0:1789760724.918806:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760724.918823:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512587392:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760724.956474:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973006 gen 3 00000008:00010000:2.0:1789760724.956529:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760724.956537:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760724.956565:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760724.957396:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760724.957436:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512587392:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760724.962333:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5687, 5687] 00000080:00200000:1.0:1789760724.962363:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760724.962370:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760724.962423:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23298048 i_blocks 47111 00000020:00200000:1.0:1789760724.962429:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760724.962450:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760724.962489:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23298048, count: 4096 00000080:00200000:1.0:1789760724.962499:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23298048, bytes: 4096 00000080:00200000:1.0:1789760724.962516:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760724.962526:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760724.962528:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23298048, 23302144) 0 0 00000080:00200000:1.0:1789760724.962534:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760724.963030:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5688, 5688] 00020000:00200000:1.0:1789760724.963042:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23298048 chunk: [23298048, 23302144) 23302144, 4096 00020000:00200000:1.0:1789760724.963047:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760724.963070:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5688, 5688] 00000008:00010000:1.0:1789760724.963307:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760724.963315:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760724.963328:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23298048, 23302144) append 0 append lockpos 0 00000080:00200000:1.0:1789760724.963343:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23298048=0x1638000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760724.963413:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760724.963417:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23302144 00000020:00010000:1.0:1789760724.963425:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760724.963429:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760724.963432:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760724.963435:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760724.963438:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760724.963447:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760724.963454:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760724.963476:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42900 00000008:00100000:1.0:1789760724.963605:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 23298048<>23302144 00000100:00100000:1.0:1789760724.963658:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760724.963686:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512587520:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760725.058372:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973007 gen 3 00000008:00010000:1.0:1789760725.058427:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42900 00000020:00200000:1.0:1789760725.058435:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760725.058447:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760725.058532:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42900 00000100:00100000:1.0:1789760725.058571:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512587520:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760725.058751:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5688, 5688] 00000080:00200000:3.0:1789760725.058768:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760725.058872:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760725.058890:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23302144 i_blocks 47111 00000020:00200000:3.0:1789760725.058894:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760725.058926:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760725.058962:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23302144, count: 4096 00000080:00200000:3.0:1789760725.058969:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23302144, bytes: 4096 00000080:00200000:3.0:1789760725.058981:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760725.058989:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760725.058992:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23302144, 23306240) 0 0 00000080:00200000:3.0:1789760725.058996:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760725.059011:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5689, 5689] 00020000:00200000:3.0:1789760725.059019:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23302144 chunk: [23302144, 23306240) 23306240, 4096 00020000:00200000:3.0:1789760725.059024:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760725.059042:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5689, 5689] 00000008:00010000:3.0:1789760725.059059:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760725.059064:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760725.059167:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23302144, 23306240) append 0 append lockpos 0 00000080:00200000:3.0:1789760725.059184:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23302144=0x1639000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760725.059244:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760725.059247:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23306240 00000020:00010000:3.0:1789760725.059255:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760725.059259:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760725.059262:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760725.059266:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760725.059268:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760725.059338:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760725.059344:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760725.059364:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760725.059470:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adf80 - object 0x240000400:5 offset 23302144<>23306240 00000100:00100000:2.0:1789760725.059846:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512587648:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760725.159432:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973008 gen 3 00000008:00010000:2.0:1789760725.159485:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760725.159494:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760725.159507:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760725.159637:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:2.0:1789760725.159670:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512587648:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760725.159760:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5689, 5689] 00000080:00200000:3.0:1789760725.159777:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760725.159783:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760725.159801:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23306240 i_blocks 47111 00000020:00200000:3.0:1789760725.159805:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760725.159815:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760725.159854:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23306240, count: 4096 00000080:00200000:3.0:1789760725.159862:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23306240, bytes: 4096 00000080:00200000:3.0:1789760725.159874:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760725.159883:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760725.159886:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23306240, 23310336) 0 0 00000080:00200000:3.0:1789760725.159890:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760725.160055:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5690, 5690] 00020000:00200000:3.0:1789760725.160062:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23306240 chunk: [23306240, 23310336) 23310336, 4096 00020000:00200000:3.0:1789760725.160067:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760725.160088:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5690, 5690] 00000008:00010000:3.0:1789760725.160106:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760725.160112:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760725.160123:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23306240, 23310336) append 0 append lockpos 0 00000080:00200000:3.0:1789760725.160135:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23306240=0x163a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760725.160187:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760725.160190:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23310336 00000020:00010000:3.0:1789760725.160198:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760725.160202:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760725.160205:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760725.160208:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760725.160211:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760725.160218:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760725.160225:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760725.160241:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760725.160348:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad880 - object 0x240000400:5 offset 23306240<>23310336 00000100:00100000:2.0:1789760725.160411:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512587776:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760725.235854:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973009 gen 3 00000008:00010000:2.0:1789760725.235941:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760725.235949:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760725.235962:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760725.235981:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760725.236010:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512587776:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760725.236237:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5690, 5690] 00000080:00200000:3.0:1789760725.236254:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760725.236261:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760725.236276:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23310336 i_blocks 47111 00000020:00200000:3.0:1789760725.236280:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760725.236290:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760725.236325:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23310336, count: 4096 00000080:00200000:3.0:1789760725.236333:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23310336, bytes: 4096 00000080:00200000:3.0:1789760725.236343:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760725.236353:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760725.236355:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23310336, 23314432) 0 0 00000080:00200000:3.0:1789760725.236360:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760725.236377:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5691, 5691] 00020000:00200000:3.0:1789760725.236386:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23310336 chunk: [23310336, 23314432) 23314432, 4096 00020000:00200000:3.0:1789760725.236391:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760725.236411:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5691, 5691] 00000008:00010000:3.0:1789760725.236429:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760725.236435:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760725.236446:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23310336, 23314432) append 0 append lockpos 0 00000080:00200000:3.0:1789760725.236459:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23310336=0x163b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760725.236518:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760725.236521:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23314432 00000020:00010000:3.0:1789760725.236529:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760725.236532:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760725.236535:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760725.236538:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760725.236541:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760725.236559:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760725.236565:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760725.236580:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760725.236677:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 23310336<>23314432 00000100:00100000:2.0:1789760725.236814:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512587904:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760725.287403:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760725.287418:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760725.287431:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760725.287498:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760725.287530:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512587904:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760725.291299:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5691, 5691] 00000080:00200000:3.0:1789760725.291327:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760725.291335:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760725.291353:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23314432 i_blocks 47111 00000020:00200000:3.0:1789760725.291358:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760725.291372:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760725.291404:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23314432, count: 4096 00000080:00200000:3.0:1789760725.291412:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23314432, bytes: 4096 00000080:00200000:3.0:1789760725.291424:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760725.291433:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760725.291435:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23314432, 23318528) 0 0 00000080:00200000:3.0:1789760725.291440:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760725.291458:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5692, 5692] 00020000:00200000:3.0:1789760725.291465:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23314432 chunk: [23314432, 23318528) 23318528, 4096 00020000:00200000:3.0:1789760725.291470:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760725.291493:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5692, 5692] 00000008:00010000:3.0:1789760725.291512:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760725.291519:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760725.291531:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23314432, 23318528) append 0 append lockpos 0 00000080:00200000:3.0:1789760725.291622:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23314432=0x163c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760725.291786:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760725.291790:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23318528 00000020:00010000:3.0:1789760725.291799:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760725.291803:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760725.291806:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760725.291810:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760725.291813:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760725.291822:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760725.291829:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760725.291861:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760725.292287:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae680 - object 0x240000400:5 offset 23314432<>23318528 00000100:00100000:2.0:1789760725.292355:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512588032:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760725.339253:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973011 gen 3 00000008:00010000:2.0:1789760725.340223:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760725.340240:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760725.340254:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760725.340287:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760725.340335:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512588032:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760725.340416:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5692, 5692] 00000080:00200000:1.0:1789760725.340432:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760725.340438:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760725.340453:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23318528 i_blocks 47111 00000020:00200000:1.0:1789760725.340457:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760725.340466:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760725.340500:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23318528, count: 4096 00000080:00200000:1.0:1789760725.340507:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23318528, bytes: 4096 00000080:00200000:1.0:1789760725.340521:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760725.340530:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760725.340532:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23318528, 23322624) 0 0 00000080:00200000:1.0:1789760725.340537:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760725.340564:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5693, 5693] 00020000:00200000:1.0:1789760725.340573:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23318528 chunk: [23318528, 23322624) 23322624, 4096 00020000:00200000:1.0:1789760725.340578:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760725.340598:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5693, 5693] 00000008:00010000:1.0:1789760725.340617:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760725.340622:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760725.340634:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23318528, 23322624) append 0 append lockpos 0 00000080:00200000:1.0:1789760725.340648:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23318528=0x163d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760725.340705:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760725.340708:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23322624 00000020:00010000:1.0:1789760725.340717:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760725.340721:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760725.340724:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760725.340728:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760725.340731:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760725.340737:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760725.340744:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760725.340761:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42900 00000008:00100000:1.0:1789760725.341063:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 23318528<>23322624 00000080:00200000:1.0:1789760725.429479:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5693, 5693] 00000080:00200000:1.0:1789760725.429495:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760725.429501:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760725.429516:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23322624 i_blocks 47111 00000020:00200000:1.0:1789760725.429520:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760725.429529:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760725.429573:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23322624, count: 4096 00000080:00200000:1.0:1789760725.429580:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23322624, bytes: 4096 00000080:00200000:1.0:1789760725.429593:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760725.429602:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760725.429605:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23322624, 23326720) 0 0 00000080:00200000:1.0:1789760725.429610:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760725.429628:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5694, 5694] 00020000:00200000:1.0:1789760725.429638:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23322624 chunk: [23322624, 23326720) 23326720, 4096 00020000:00200000:1.0:1789760725.429642:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760725.429662:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5694, 5694] 00000008:00010000:1.0:1789760725.429680:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760725.429686:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760725.429699:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23322624, 23326720) append 0 append lockpos 0 00000080:00200000:1.0:1789760725.429711:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23322624=0x163e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760725.429766:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760725.429768:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23326720 00000020:00010000:1.0:1789760725.429777:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760725.429780:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760725.429784:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760725.429787:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760725.429790:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760725.432014:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760725.432027:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760725.432066:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f429c0 00000008:00100000:1.0:1789760725.432186:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 23322624<>23326720 00000080:00200000:1.0:1789760725.490113:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5694, 5694] 00000080:00200000:1.0:1789760725.490130:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760725.490136:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760725.490151:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23326720 i_blocks 47111 00000020:00200000:1.0:1789760725.490154:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760725.490164:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760725.490199:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23326720, count: 4096 00000080:00200000:1.0:1789760725.490206:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23326720, bytes: 4096 00000080:00200000:1.0:1789760725.490219:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760725.490228:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760725.490230:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23326720, 23330816) 0 0 00000080:00200000:1.0:1789760725.490235:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760725.490254:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5695, 5695] 00020000:00200000:1.0:1789760725.490263:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23326720 chunk: [23326720, 23330816) 23330816, 4096 00020000:00200000:1.0:1789760725.490267:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760725.490286:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5695, 5695] 00000008:00010000:1.0:1789760725.490305:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760725.490311:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760725.490322:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23326720, 23330816) append 0 append lockpos 0 00000080:00200000:1.0:1789760725.490335:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23326720=0x163f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760725.490399:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760725.490402:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23330816 00000020:00010000:1.0:1789760725.490410:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760725.490414:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760725.490417:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760725.490420:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760725.490423:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760725.490431:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760725.490438:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760725.490455:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760725.491231:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97100 - object 0x240000400:5 offset 23326720<>23330816 00000080:00200000:1.0:1789760725.534360:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5695, 5695] 00000080:00200000:1.0:1789760725.534389:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760725.534396:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760725.534413:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23330816 i_blocks 47111 00000020:00200000:1.0:1789760725.534417:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760725.534427:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760725.534464:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23330816, count: 4096 00000080:00200000:1.0:1789760725.534471:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23330816, bytes: 4096 00000080:00200000:1.0:1789760725.534482:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760725.534491:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760725.534493:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23330816, 23334912) 0 0 00000080:00200000:1.0:1789760725.534498:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760725.534516:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5696, 5696] 00020000:00200000:1.0:1789760725.534525:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23330816 chunk: [23330816, 23334912) 23334912, 4096 00020000:00200000:1.0:1789760725.534530:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760725.534731:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5696, 5696] 00000008:00010000:1.0:1789760725.534751:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760725.534757:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760725.534769:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23330816, 23334912) append 0 append lockpos 0 00000080:00200000:1.0:1789760725.534781:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23330816=0x1640000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760725.534838:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760725.534840:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23334912 00000020:00010000:1.0:1789760725.534849:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760725.534852:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760725.534855:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760725.534859:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760725.534862:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760725.534869:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760725.534876:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760725.534893:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760725.535119:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 23330816<>23334912 00000080:00200000:2.0:1789760725.590058:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5696, 5696] 00000080:00200000:2.0:1789760725.590076:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760725.590082:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760725.590099:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23334912 i_blocks 47111 00000020:00200000:2.0:1789760725.590103:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760725.590113:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760725.590148:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23334912, count: 4096 00000080:00200000:2.0:1789760725.590156:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23334912, bytes: 4096 00000080:00200000:2.0:1789760725.590167:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760725.590176:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760725.590178:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23334912, 23339008) 0 0 00000080:00200000:2.0:1789760725.590183:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760725.590201:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5697, 5697] 00020000:00200000:2.0:1789760725.590210:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23334912 chunk: [23334912, 23339008) 23339008, 4096 00020000:00200000:2.0:1789760725.590215:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760725.590236:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5697, 5697] 00000008:00010000:2.0:1789760725.590256:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760725.590261:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760725.590272:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23334912, 23339008) append 0 append lockpos 0 00000080:00200000:2.0:1789760725.590285:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23334912=0x1641000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760725.590336:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760725.590338:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23339008 00000020:00010000:2.0:1789760725.590346:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760725.590350:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.590353:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.590356:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760725.590359:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760725.590366:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760725.590437:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760725.590456:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760725.590627:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 23334912<>23339008 00000100:00100000:3.0:1789760725.590702:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512588672:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760725.656115:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973016 gen 3 00000008:00010000:2.0:1789760725.656168:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760725.656176:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760725.656187:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760725.656212:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760725.656244:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512588672:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760725.657417:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512588800:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760725.725573:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973017 gen 3 00000008:00010000:1.0:1789760725.725748:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760725.725756:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760725.725767:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760725.725786:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760725.725818:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512588800:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760725.725943:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5698, 5698] 00000080:00200000:2.0:1789760725.725960:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760725.725966:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760725.725983:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23343104 i_blocks 47111 00000020:00200000:2.0:1789760725.725986:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760725.725996:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760725.726035:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23343104, count: 4096 00000080:00200000:2.0:1789760725.726043:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23343104, bytes: 4096 00000080:00200000:2.0:1789760725.726053:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760725.726062:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760725.726064:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23343104, 23347200) 0 0 00000080:00200000:2.0:1789760725.726068:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760725.726088:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5699, 5699] 00020000:00200000:2.0:1789760725.726101:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23343104 chunk: [23343104, 23347200) 23347200, 4096 00020000:00200000:2.0:1789760725.726105:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760725.726126:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5699, 5699] 00000008:00010000:2.0:1789760725.726143:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760725.726150:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760725.726160:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23343104, 23347200) append 0 append lockpos 0 00000080:00200000:2.0:1789760725.726173:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23343104=0x1643000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760725.726338:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760725.726341:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23347200 00000020:00010000:2.0:1789760725.726350:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760725.726354:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.726357:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42c68/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.726360:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760725.726363:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760725.726370:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760725.726376:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760725.726393:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760725.726522:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 23343104<>23347200 00000100:00100000:3.0:1789760725.726748:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512588928:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760725.789395:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760725.807436:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973018 gen 3 00000008:00010000:2.0:1789760725.807489:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760725.807497:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760725.807508:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760725.807534:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760725.807575:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512588928:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760725.809641:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512589056:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760725.887390:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973019 gen 3 00000008:00010000:1.0:1789760725.887446:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760725.887454:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760725.887466:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760725.887552:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760725.887586:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512589056:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760725.892915:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23351296 i_blocks 47111 00000020:00200000:1.0:1789760725.892985:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760725.893006:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760725.893041:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23351296, count: 4096 00000080:00200000:1.0:1789760725.893050:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23351296, bytes: 4096 00000080:00200000:1.0:1789760725.893066:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760725.893075:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760725.893077:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23351296, 23355392) 0 0 00000080:00200000:1.0:1789760725.893081:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760725.893098:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5701, 5701] 00020000:00200000:1.0:1789760725.893107:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23351296 chunk: [23351296, 23355392) 23355392, 4096 00020000:00200000:1.0:1789760725.893111:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760725.893134:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5701, 5701] 00000008:00010000:1.0:1789760725.893154:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760725.893160:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760725.893172:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23351296, 23355392) append 0 append lockpos 0 00000080:00200000:1.0:1789760725.893187:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23351296=0x1645000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760725.893255:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760725.893258:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23355392 00000020:00010000:1.0:1789760725.893266:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760725.893269:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760725.893272:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760725.893276:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760725.893279:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760725.893289:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760725.893296:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760725.893349:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f429c0 00000008:00100000:1.0:1789760725.893467:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 23351296<>23355392 00000100:00100000:1.0:1789760725.983405:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973020 gen 3 00000008:00010000:1.0:1789760725.983460:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f429c0 00000020:00200000:1.0:1789760725.983470:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760725.983481:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760725.983504:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f429c0 00000100:00100000:1.0:1789760725.983535:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512589184:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760725.983656:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5701, 5701] 00000080:00200000:2.0:1789760725.983672:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760725.983679:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760725.983695:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23355392 i_blocks 47111 00000020:00200000:2.0:1789760725.983699:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760725.983709:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760725.983752:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23355392, count: 4096 00000080:00200000:2.0:1789760725.983759:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23355392, bytes: 4096 00000080:00200000:2.0:1789760725.983778:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760725.983788:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760725.983790:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23355392, 23359488) 0 0 00000080:00200000:2.0:1789760725.983795:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760725.983812:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5702, 5702] 00020000:00200000:2.0:1789760725.983820:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23355392 chunk: [23355392, 23359488) 23359488, 4096 00020000:00200000:2.0:1789760725.983824:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760725.983844:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5702, 5702] 00000008:00010000:2.0:1789760725.983862:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760725.983868:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760725.983880:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23355392, 23359488) append 0 append lockpos 0 00000080:00200000:2.0:1789760725.983897:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23355392=0x1646000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760725.983990:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760725.983993:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23359488 00000020:00010000:2.0:1789760725.984002:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760725.984005:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.984008:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760725.984012:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760725.984015:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760725.984023:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760725.984030:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760725.984047:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760725.984153:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 23355392<>23359488 00000100:00100000:2.0:1789760725.984203:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760725.984219:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512589312:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760726.059291:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760726.059308:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760726.059320:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760726.059350:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760726.059383:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512589312:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760726.059632:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5702, 5702] 00000080:00200000:1.0:1789760726.059649:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760726.059654:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760726.059669:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23359488 i_blocks 47111 00000020:00200000:1.0:1789760726.059673:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760726.059684:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760726.059725:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23359488, count: 4096 00000080:00200000:1.0:1789760726.059732:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23359488, bytes: 4096 00000080:00200000:1.0:1789760726.059744:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760726.059753:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760726.059755:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23359488, 23363584) 0 0 00000080:00200000:1.0:1789760726.059760:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760726.059778:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5703, 5703] 00020000:00200000:1.0:1789760726.059786:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23359488 chunk: [23359488, 23363584) 23363584, 4096 00020000:00200000:1.0:1789760726.059791:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760726.059812:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5703, 5703] 00000008:00010000:1.0:1789760726.060787:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760726.060800:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760726.060820:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23359488, 23363584) append 0 append lockpos 0 00000080:00200000:1.0:1789760726.060852:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23359488=0x1647000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760726.061561:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760726.061567:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23363584 00000020:00010000:1.0:1789760726.061578:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760726.061584:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760726.061588:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760726.061591:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760726.061594:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760726.061603:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760726.061612:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760726.061633:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f429c0 00000008:00100000:1.0:1789760726.061765:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 23359488<>23363584 00000080:00200000:1.0:1789760726.114817:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5703, 5703] 00000080:00200000:1.0:1789760726.114835:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760726.114842:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760726.114857:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23363584 i_blocks 47111 00000020:00200000:1.0:1789760726.114861:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760726.114872:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760726.114951:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23363584, count: 4096 00000080:00200000:1.0:1789760726.114959:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23363584, bytes: 4096 00000080:00200000:1.0:1789760726.114970:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760726.114981:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760726.114983:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23363584, 23367680) 0 0 00000080:00200000:1.0:1789760726.114986:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760726.115003:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5704, 5704] 00020000:00200000:1.0:1789760726.115012:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23363584 chunk: [23363584, 23367680) 23367680, 4096 00020000:00200000:1.0:1789760726.115015:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760726.115035:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5704, 5704] 00000008:00010000:1.0:1789760726.115053:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760726.115059:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760726.115071:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23363584, 23367680) append 0 append lockpos 0 00000080:00200000:1.0:1789760726.115085:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23363584=0x1648000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760726.115141:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760726.115144:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23367680 00000020:00010000:1.0:1789760726.115152:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760726.115156:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760726.115158:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760726.115161:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760726.115164:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760726.115172:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760726.115178:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760726.115196:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:1.0:1789760726.115295:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97100 - object 0x240000400:5 offset 23363584<>23367680 00000080:00200000:2.0:1789760726.185661:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5704, 5704] 00000080:00200000:2.0:1789760726.185678:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760726.185684:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760726.185699:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23367680 i_blocks 47111 00000020:00200000:2.0:1789760726.185703:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760726.185713:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760726.185748:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23367680, count: 4096 00000080:00200000:2.0:1789760726.185756:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23367680, bytes: 4096 00000080:00200000:2.0:1789760726.185768:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760726.185780:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760726.185784:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23367680, 23371776) 0 0 00000080:00200000:2.0:1789760726.185789:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760726.186027:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5705, 5705] 00020000:00200000:2.0:1789760726.186038:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23367680 chunk: [23367680, 23371776) 23371776, 4096 00020000:00200000:2.0:1789760726.186043:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760726.186063:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5705, 5705] 00000008:00010000:2.0:1789760726.186083:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760726.186088:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760726.186100:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23367680, 23371776) append 0 append lockpos 0 00000080:00200000:2.0:1789760726.186115:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23367680=0x1649000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760726.186175:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760726.186179:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23371776 00000020:00010000:2.0:1789760726.186187:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760726.186191:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760726.186195:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760726.186199:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760726.186202:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760726.186210:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760726.186218:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760726.186237:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760726.186349:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 23367680<>23371776 00000100:00100000:3.0:1789760726.186432:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512589696:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760726.256944:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973024 gen 3 00000008:00010000:2.0:1789760726.257014:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760726.257024:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760726.257035:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760726.257052:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760726.257082:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512589696:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760726.317388:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5706, 5706] 00000080:00200000:2.0:1789760726.317406:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760726.317413:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760726.317428:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23375872 i_blocks 47111 00000020:00200000:2.0:1789760726.317432:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760726.317453:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760726.317487:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23375872, count: 4096 00000080:00200000:2.0:1789760726.317495:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23375872, bytes: 4096 00000080:00200000:2.0:1789760726.317509:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760726.317518:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760726.317520:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23375872, 23379968) 0 0 00000080:00200000:2.0:1789760726.317534:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760726.317560:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5707, 5707] 00020000:00200000:2.0:1789760726.317570:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23375872 chunk: [23375872, 23379968) 23379968, 4096 00020000:00200000:2.0:1789760726.317576:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760726.317606:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5707, 5707] 00000008:00010000:2.0:1789760726.317625:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760726.317632:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760726.317644:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23375872, 23379968) append 0 append lockpos 0 00000080:00200000:2.0:1789760726.317658:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23375872=0x164b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760726.317718:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760726.317721:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23379968 00000020:00010000:2.0:1789760726.317729:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760726.317733:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760726.317736:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760726.317740:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760726.317743:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760726.317750:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760726.317757:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760726.317774:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760726.317880:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 23375872<>23379968 00000100:00100000:2.0:1789760726.317972:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760726.317989:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512589952:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760726.358542:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760726.358559:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760726.358571:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760726.358651:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760726.358683:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512589952:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760726.363431:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760726.363459:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512590080:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760726.415805:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973026 gen 3 00000100:00100000:1.0:1789760726.415850:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005148700 x1876699512590080/t4294973027(4294973027) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760742 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760726.415948:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760726.415956:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760726.415967:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760726.415989:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760726.416013:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512590080:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760726.417941:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512590208:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760726.489203:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973028 gen 3 00000008:00010000:1.0:1789760726.489272:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760726.489281:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760726.489292:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760726.489304:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760726.489334:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512590208:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760726.490251:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512590336:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760726.556777:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973029 gen 3 00000008:00010000:1.0:1789760726.556878:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760726.556889:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760726.557003:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760726.557026:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760726.557055:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512590336:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760726.560766:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512590464:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760726.645689:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760726.645705:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760726.645720:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760726.645826:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760726.645858:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512590464:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760726.649213:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760726.649242:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512590592:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760726.719114:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973031 gen 3 00000008:00010000:1.0:1789760726.719198:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760726.719208:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760726.719220:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760726.719247:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760726.719279:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512590592:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760726.720487:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512590720:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760726.755850:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973032 gen 3 00000008:00010000:1.0:1789760726.755886:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760726.755893:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760726.755938:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760726.755954:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760726.755977:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512590720:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760726.757870:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512590848:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760726.806091:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760726.806107:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760726.806121:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760726.806153:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760726.806190:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512590848:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760726.809323:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5714, 5714] 00000080:00200000:3.0:1789760726.809351:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760726.809358:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760726.809376:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23408640 i_blocks 47111 00000020:00200000:3.0:1789760726.809382:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760726.809401:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760726.809709:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23408640, count: 4096 00000080:00200000:3.0:1789760726.809721:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23408640, bytes: 4096 00000080:00200000:3.0:1789760726.809737:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760726.809746:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760726.809748:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23408640, 23412736) 0 0 00000080:00200000:3.0:1789760726.809754:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760726.809772:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5715, 5715] 00020000:00200000:3.0:1789760726.809783:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23408640 chunk: [23408640, 23412736) 23412736, 4096 00020000:00200000:3.0:1789760726.809788:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760726.809811:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5715, 5715] 00000008:00010000:3.0:1789760726.809948:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760726.809955:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760726.809970:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23408640, 23412736) append 0 append lockpos 0 00000080:00200000:3.0:1789760726.809985:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23408640=0x1653000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760726.810317:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760726.810321:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23412736 00000020:00010000:3.0:1789760726.810330:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760726.810334:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760726.810337:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760726.810340:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760726.810343:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760726.810351:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760726.810358:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760726.810385:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760726.810500:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 23408640<>23412736 00000100:00100000:2.0:1789760726.810603:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512590976:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760726.858738:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973034 gen 3 00000008:00010000:2.0:1789760726.858819:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760726.858839:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760726.858852:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760726.858874:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760726.858947:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512590976:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760726.858968:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5715, 5715] 00000080:00200000:3.0:1789760726.858994:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760726.859000:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760726.859017:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23412736 i_blocks 47111 00000020:00200000:3.0:1789760726.859022:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760726.859037:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760726.859062:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23412736, count: 4096 00000080:00200000:3.0:1789760726.859070:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23412736, bytes: 4096 00000080:00200000:3.0:1789760726.859082:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760726.859089:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760726.859091:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23412736, 23416832) 0 0 00000080:00200000:3.0:1789760726.859094:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760726.859107:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5716, 5716] 00020000:00200000:3.0:1789760726.859115:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23412736 chunk: [23412736, 23416832) 23416832, 4096 00020000:00200000:3.0:1789760726.859120:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760726.859137:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5716, 5716] 00000008:00010000:3.0:1789760726.859153:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760726.859159:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760726.859170:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23412736, 23416832) append 0 append lockpos 0 00000080:00200000:3.0:1789760726.859182:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23412736=0x1654000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760726.860726:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760726.860733:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23416832 00000020:00010000:3.0:1789760726.860745:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760726.860750:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760726.860753:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760726.860756:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760726.860760:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760726.860769:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760726.860777:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760726.860800:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760726.862024:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af800 - object 0x240000400:5 offset 23412736<>23416832 00000100:00100000:3.0:1789760726.865063:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760726.865110:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512591104:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760726.907749:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973035 gen 3 00000008:00010000:2.0:1789760726.907798:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760726.907807:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760726.907821:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760726.907849:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:2.0:1789760726.909003:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512591104:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760726.909063:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc300 00000008:00100000:2.0:1789760726.909770:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 23416832<>23420928 00000100:00100000:2.0:1789760726.909837:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512591232:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760726.973429:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973036 gen 3 00000008:00010000:2.0:1789760726.973463:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:2.0:1789760726.973469:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760726.973478:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760726.973498:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:2.0:1789760726.973520:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512591232:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760726.975067:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512591360:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760727.023051:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760727.023062:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760727.023071:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760727.023092:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760727.023117:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512591360:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760727.024094:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760727.024112:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512591488:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760727.059318:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973038 gen 3 00000008:00010000:1.0:1789760727.059363:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760727.059379:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760727.059387:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760727.059439:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760727.059458:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512591488:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760727.060053:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512591616:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760727.107755:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760727.107766:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760727.107777:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760727.107801:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760727.107827:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512591616:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760727.107985:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5720, 5720] 00000080:00200000:2.0:1789760727.107998:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760727.108003:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760727.108013:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23433216 i_blocks 47111 00000020:00200000:2.0:1789760727.108016:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760727.108024:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760727.108051:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23433216, count: 4096 00000080:00200000:2.0:1789760727.108057:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23433216, bytes: 4096 00000080:00200000:2.0:1789760727.108065:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760727.108073:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760727.108075:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23433216, 23437312) 0 0 00000080:00200000:2.0:1789760727.108078:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760727.108092:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5721, 5721] 00020000:00200000:2.0:1789760727.108100:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23433216 chunk: [23433216, 23437312) 23437312, 4096 00020000:00200000:2.0:1789760727.108104:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760727.108120:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5721, 5721] 00000008:00010000:2.0:1789760727.108134:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760727.108138:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760727.108147:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23433216, 23437312) append 0 append lockpos 0 00000080:00200000:2.0:1789760727.108157:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23433216=0x1659000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760727.108200:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760727.108203:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23437312 00000020:00010000:2.0:1789760727.108276:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760727.108279:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760727.108282:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760727.108286:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760727.108289:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760727.108296:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760727.108302:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760727.108316:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760727.108392:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 23433216<>23437312 00000100:00100000:3.0:1789760727.108455:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512591744:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760727.145864:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973039 gen 3 00000100:00100000:2.0:1789760727.145897:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368d880 x1876699512591744/t4294973040(4294973040) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760743 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760727.145969:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760727.145974:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760727.145981:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760727.146000:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760727.146019:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512591744:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760727.146236:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5721, 5721] 00000080:00200000:1.0:1789760727.146250:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760727.146254:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760727.146265:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23437312 i_blocks 47111 00000020:00200000:1.0:1789760727.146268:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760727.146276:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760727.146300:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23437312, count: 4096 00000080:00200000:1.0:1789760727.146306:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23437312, bytes: 4096 00000080:00200000:1.0:1789760727.146314:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760727.146321:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760727.146323:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23437312, 23441408) 0 0 00000080:00200000:1.0:1789760727.146327:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760727.146340:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5722, 5722] 00020000:00200000:1.0:1789760727.146347:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23437312 chunk: [23437312, 23441408) 23441408, 4096 00020000:00200000:1.0:1789760727.146351:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760727.146366:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5722, 5722] 00000008:00010000:1.0:1789760727.146379:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760727.146384:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760727.146394:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23437312, 23441408) append 0 append lockpos 0 00000080:00200000:1.0:1789760727.146404:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23437312=0x165a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760727.146448:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760727.146451:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23441408 00000020:00010000:1.0:1789760727.146457:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760727.146460:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.146463:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.146465:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760727.146468:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760727.146475:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760727.146480:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760727.146492:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760727.146578:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 23437312<>23441408 00000080:00200000:1.0:1789760727.215937:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5722, 5722] 00000080:00200000:1.0:1789760727.215950:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760727.215954:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760727.215965:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23441408 i_blocks 47111 00000020:00200000:1.0:1789760727.215968:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760727.215975:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760727.216001:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23441408, count: 4096 00000080:00200000:1.0:1789760727.216007:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23441408, bytes: 4096 00000080:00200000:1.0:1789760727.216014:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760727.216020:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760727.216022:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23441408, 23445504) 0 0 00000080:00200000:1.0:1789760727.216026:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760727.216037:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5723, 5723] 00020000:00200000:1.0:1789760727.216044:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23441408 chunk: [23441408, 23445504) 23445504, 4096 00020000:00200000:1.0:1789760727.216048:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760727.216062:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5723, 5723] 00000008:00010000:1.0:1789760727.216074:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760727.216079:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760727.216087:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23441408, 23445504) append 0 append lockpos 0 00000080:00200000:1.0:1789760727.216096:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23441408=0x165b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760727.216136:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760727.216138:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23445504 00000020:00010000:1.0:1789760727.216143:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760727.216146:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.216149:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.216153:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760727.216155:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760727.216161:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760727.216166:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760727.216178:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760727.216415:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 23441408<>23445504 00000100:00100000:1.0:1789760727.216458:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760727.216470:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512592000:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760727.256567:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973042 gen 3 00000008:00010000:1.0:1789760727.256616:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760727.256622:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760727.256629:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760727.256653:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760727.256676:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512592000:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760727.256959:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5723, 5723] 00000080:00200000:2.0:1789760727.256972:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760727.256978:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760727.256991:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23445504 i_blocks 47111 00000020:00200000:2.0:1789760727.256995:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760727.257003:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760727.257030:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23445504, count: 4096 00000080:00200000:2.0:1789760727.257036:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23445504, bytes: 4096 00000080:00200000:2.0:1789760727.257045:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760727.257052:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760727.257054:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23445504, 23449600) 0 0 00000080:00200000:2.0:1789760727.257058:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760727.257071:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5724, 5724] 00020000:00200000:2.0:1789760727.257077:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23445504 chunk: [23445504, 23449600) 23449600, 4096 00020000:00200000:2.0:1789760727.257082:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760727.257095:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5724, 5724] 00000008:00010000:2.0:1789760727.257110:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760727.257115:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760727.257126:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23445504, 23449600) append 0 append lockpos 0 00000080:00200000:2.0:1789760727.257137:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23445504=0x165c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760727.257183:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760727.257186:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23449600 00000020:00010000:2.0:1789760727.257192:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760727.257195:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760727.257197:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760727.257201:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760727.257216:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760727.257224:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760727.257230:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760727.257245:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760727.257326:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 23445504<>23449600 00000100:00100000:2.0:1789760727.257367:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760727.257383:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512592128:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760727.306955:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973043 gen 3 00000008:00010000:2.0:1789760727.306993:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760727.306999:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760727.307008:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760727.307081:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760727.307104:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512592128:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760727.307402:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5724, 5724] 00000080:00200000:1.0:1789760727.307415:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760727.307419:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760727.307431:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23449600 i_blocks 47111 00000020:00200000:1.0:1789760727.307434:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760727.307443:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760727.307712:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23449600, count: 4096 00000080:00200000:1.0:1789760727.307720:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23449600, bytes: 4096 00000080:00200000:1.0:1789760727.307730:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760727.307737:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760727.307739:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23449600, 23453696) 0 0 00000080:00200000:1.0:1789760727.307743:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760727.307756:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5725, 5725] 00020000:00200000:1.0:1789760727.307764:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23449600 chunk: [23449600, 23453696) 23453696, 4096 00020000:00200000:1.0:1789760727.307768:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760727.307783:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5725, 5725] 00000008:00010000:1.0:1789760727.307797:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760727.307802:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760727.307810:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23449600, 23453696) append 0 append lockpos 0 00000080:00200000:1.0:1789760727.307821:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23449600=0x165d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760727.307867:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760727.307870:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23453696 00000020:00010000:1.0:1789760727.307876:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760727.307879:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.307882:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760727.307885:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760727.307888:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760727.307894:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760727.307939:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760727.307955:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760727.308029:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 23449600<>23453696 00000080:00200000:2.0:1789760727.341487:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5725, 5725] 00000080:00200000:2.0:1789760727.341499:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760727.341504:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760727.341515:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23453696 i_blocks 47111 00000020:00200000:2.0:1789760727.341518:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760727.341526:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760727.341565:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23453696, count: 4096 00000080:00200000:2.0:1789760727.341572:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23453696, bytes: 4096 00000080:00200000:2.0:1789760727.341579:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760727.341586:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760727.341588:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23453696, 23457792) 0 0 00000080:00200000:2.0:1789760727.341592:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760727.341605:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5726, 5726] 00020000:00200000:2.0:1789760727.341612:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23453696 chunk: [23453696, 23457792) 23457792, 4096 00020000:00200000:2.0:1789760727.341616:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760727.341734:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5726, 5726] 00000008:00010000:2.0:1789760727.341750:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760727.341755:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760727.341765:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23453696, 23457792) append 0 append lockpos 0 00000080:00200000:2.0:1789760727.341775:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23453696=0x165e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760727.341821:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760727.341824:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23457792 00000020:00010000:2.0:1789760727.341830:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760727.341833:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760727.341836:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760727.341839:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760727.341841:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760727.341847:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760727.341852:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760727.341865:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760727.342766:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 23453696<>23457792 00000100:00100000:2.0:1789760727.342829:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512592384:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760727.391202:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973045 gen 3 00000008:00010000:2.0:1789760727.391239:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760727.391246:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760727.391255:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760727.391277:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760727.391302:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512592384:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760727.391335:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5726, 5726] 00000080:00200000:3.0:1789760727.391361:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760727.391370:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760727.391391:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23457792 i_blocks 47111 00000020:00200000:3.0:1789760727.391394:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760727.391406:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760727.391434:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23457792, count: 4096 00000080:00200000:3.0:1789760727.391441:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23457792, bytes: 4096 00000080:00200000:3.0:1789760727.391453:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760727.391459:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760727.391461:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23457792, 23461888) 0 0 00000080:00200000:3.0:1789760727.391465:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760727.391482:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5727, 5727] 00020000:00200000:3.0:1789760727.391490:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23457792 chunk: [23457792, 23461888) 23461888, 4096 00020000:00200000:3.0:1789760727.391493:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760727.391509:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5727, 5727] 00000008:00010000:3.0:1789760727.391522:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760727.391526:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760727.391546:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23457792, 23461888) append 0 append lockpos 0 00000080:00200000:3.0:1789760727.391556:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23457792=0x165f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760727.391597:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760727.391612:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23461888 00000020:00010000:3.0:1789760727.391627:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760727.391630:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.391633:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.391637:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760727.391639:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760727.391645:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760727.391650:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760727.391671:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760727.391754:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af800 - object 0x240000400:5 offset 23457792<>23461888 00000100:00100000:2.0:1789760727.393657:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512592512:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760727.468125:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973046 gen 3 00000008:00010000:2.0:1789760727.468161:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760727.468167:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760727.468179:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760727.468207:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760727.468233:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512592512:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760727.469309:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5727, 5727] 00000080:00200000:3.0:1789760727.469327:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760727.469332:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760727.469344:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23461888 i_blocks 47111 00000020:00200000:3.0:1789760727.469348:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760727.469358:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760727.469383:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23461888, count: 4096 00000080:00200000:3.0:1789760727.469389:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23461888, bytes: 4096 00000080:00200000:3.0:1789760727.469399:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760727.469408:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760727.469410:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23461888, 23465984) 0 0 00000080:00200000:3.0:1789760727.469414:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760727.469427:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5728, 5728] 00020000:00200000:3.0:1789760727.469435:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23461888 chunk: [23461888, 23465984) 23465984, 4096 00020000:00200000:3.0:1789760727.469438:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760727.469453:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5728, 5728] 00000008:00010000:3.0:1789760727.469466:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760727.469471:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760727.469479:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23461888, 23465984) append 0 append lockpos 0 00000080:00200000:3.0:1789760727.469490:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23461888=0x1660000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760727.469554:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760727.469557:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23465984 00000020:00010000:3.0:1789760727.469563:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760727.469566:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.469569:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.469572:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760727.469574:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760727.469581:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760727.469587:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760727.469601:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760727.469686:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 23461888<>23465984 00000100:00100000:2.0:1789760727.470161:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760727.470182:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512592640:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760727.507458:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973047 gen 3 00000008:00010000:2.0:1789760727.507494:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760727.507500:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760727.507508:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760727.507535:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760727.507559:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512592640:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760727.507644:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5728, 5728] 00000080:00200000:3.0:1789760727.507659:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760727.507663:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760727.507674:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23465984 i_blocks 47111 00000020:00200000:3.0:1789760727.507678:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760727.507685:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760727.507711:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23465984, count: 4096 00000080:00200000:3.0:1789760727.507717:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23465984, bytes: 4096 00000080:00200000:3.0:1789760727.507726:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760727.507732:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760727.507734:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23465984, 23470080) 0 0 00000080:00200000:3.0:1789760727.507738:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760727.507751:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5729, 5729] 00020000:00200000:3.0:1789760727.507758:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23465984 chunk: [23465984, 23470080) 23470080, 4096 00020000:00200000:3.0:1789760727.507761:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760727.507776:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5729, 5729] 00000008:00010000:3.0:1789760727.507791:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760727.507796:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760727.507804:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23465984, 23470080) append 0 append lockpos 0 00000080:00200000:3.0:1789760727.507816:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23465984=0x1661000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760727.507858:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760727.507861:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23470080 00000020:00010000:3.0:1789760727.507868:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760727.507871:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.507874:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.507878:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760727.507880:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760727.507886:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760727.507892:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760727.507954:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760727.508036:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae680 - object 0x240000400:5 offset 23465984<>23470080 00000100:00100000:2.0:1789760727.508160:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512592768:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760727.543216:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973048 gen 3 00000008:00010000:2.0:1789760727.543253:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760727.543259:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760727.543268:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760727.543287:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760727.543310:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512592768:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760727.543775:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5729, 5729] 00000080:00200000:3.0:1789760727.543788:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760727.543794:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760727.543806:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23470080 i_blocks 47111 00000020:00200000:3.0:1789760727.543810:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760727.543817:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760727.543845:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23470080, count: 4096 00000080:00200000:3.0:1789760727.543851:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23470080, bytes: 4096 00000080:00200000:3.0:1789760727.543860:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760727.543869:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760727.543871:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23470080, 23474176) 0 0 00000080:00200000:3.0:1789760727.543874:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760727.543887:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5730, 5730] 00020000:00200000:3.0:1789760727.543894:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23470080 chunk: [23470080, 23474176) 23474176, 4096 00020000:00200000:3.0:1789760727.543897:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760727.543960:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5730, 5730] 00000008:00010000:3.0:1789760727.543975:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760727.543980:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760727.543989:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23470080, 23474176) append 0 append lockpos 0 00000080:00200000:3.0:1789760727.544000:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23470080=0x1662000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760727.544045:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760727.544047:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23474176 00000020:00010000:3.0:1789760727.544054:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760727.544057:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.544060:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.544063:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760727.544066:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760727.544072:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760727.544079:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760727.544095:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760727.544420:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 23470080<>23474176 00000100:00100000:3.0:1789760727.544461:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760727.544478:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512592896:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760727.593209:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760727.593220:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760727.593229:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760727.593240:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760727.593263:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512592896:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760727.594942:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512593024:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760727.641825:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973050 gen 3 00000008:00010000:1.0:1789760727.641877:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760727.641884:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760727.641893:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760727.641994:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760727.642019:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512593024:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760727.643405:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512593152:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760727.684872:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973051 gen 3 00000008:00010000:1.0:1789760727.684933:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760727.684939:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760727.684948:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760727.684964:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760727.684985:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512593152:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760727.685760:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512593280:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760727.737213:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973052 gen 3 00000008:00010000:1.0:1789760727.737246:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760727.737252:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760727.737261:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760727.737288:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760727.737309:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512593280:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760727.739415:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512593408:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760727.812650:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973053 gen 3 00000008:00010000:1.0:1789760727.812689:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760727.812697:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760727.812706:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760727.812727:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760727.812751:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512593408:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760727.813485:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512593536:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760727.853563:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973054 gen 3 00000008:00010000:1.0:1789760727.853599:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760727.853605:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760727.853613:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760727.853636:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760727.853657:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512593536:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760727.854704:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760727.854723:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512593664:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760727.895665:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973055 gen 3 00000008:00010000:1.0:1789760727.895765:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760727.895772:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760727.895781:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760727.895853:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760727.895878:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512593664:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760727.896650:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512593792:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760727.950034:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973056 gen 3 00000008:00010000:1.0:1789760727.950068:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760727.950074:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760727.950082:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760727.950142:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760727.950164:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512593792:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760727.950509:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5737, 5737] 00000080:00200000:3.0:1789760727.950521:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760727.950535:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760727.950546:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23502848 i_blocks 47111 00000020:00200000:3.0:1789760727.950550:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760727.950558:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760727.950581:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23502848, count: 4096 00000080:00200000:3.0:1789760727.950586:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23502848, bytes: 4096 00000080:00200000:3.0:1789760727.950594:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760727.950600:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760727.950602:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23502848, 23506944) 0 0 00000080:00200000:3.0:1789760727.950606:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760727.950617:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5738, 5738] 00020000:00200000:3.0:1789760727.950623:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23502848 chunk: [23502848, 23506944) 23506944, 4096 00020000:00200000:3.0:1789760727.950627:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760727.950640:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5738, 5738] 00000008:00010000:3.0:1789760727.950651:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760727.950656:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760727.950664:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23502848, 23506944) append 0 append lockpos 0 00000080:00200000:3.0:1789760727.950673:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23502848=0x166a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760727.950712:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760727.950714:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23506944 00000020:00010000:3.0:1789760727.950720:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760727.950723:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.950725:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760727.950729:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760727.950731:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760727.950737:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760727.950741:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760727.950754:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760727.950875:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad880 - object 0x240000400:5 offset 23502848<>23506944 00000100:00100000:2.0:1789760727.950965:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512593920:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760728.000069:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973057 gen 3 00000008:00010000:2.0:1789760728.000103:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760728.000109:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760728.000118:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760728.000137:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760728.000159:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512593920:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760728.000246:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5738, 5738] 00000080:00200000:3.0:1789760728.000257:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.000262:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.000272:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23506944 i_blocks 47111 00000020:00200000:3.0:1789760728.000276:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.000294:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.000319:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23506944, count: 4096 00000080:00200000:3.0:1789760728.000325:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23506944, bytes: 4096 00000080:00200000:3.0:1789760728.000332:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.000340:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.000341:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23506944, 23511040) 0 0 00000080:00200000:3.0:1789760728.000345:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.000357:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5739, 5739] 00020000:00200000:3.0:1789760728.000364:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23506944 chunk: [23506944, 23511040) 23511040, 4096 00020000:00200000:3.0:1789760728.000367:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.000380:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5739, 5739] 00000008:00010000:3.0:1789760728.000393:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.000397:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760728.000407:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23506944, 23511040) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.000418:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23506944=0x166b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.000460:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.000462:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23511040 00000020:00010000:3.0:1789760728.000469:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.000472:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.000474:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.000477:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.000479:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.000485:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.000490:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.000504:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760728.000593:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adf80 - object 0x240000400:5 offset 23506944<>23511040 00000100:00100000:2.0:1789760728.000858:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760728.000872:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512594048:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760728.040540:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973058 gen 3 00000008:00010000:2.0:1789760728.040577:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760728.040583:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760728.040592:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760728.040610:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760728.040629:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512594048:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760728.040830:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5739, 5739] 00000080:00200000:3.0:1789760728.040846:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.040860:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.040873:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23511040 i_blocks 47111 00000020:00200000:3.0:1789760728.040876:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.040884:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.040939:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23511040, count: 4096 00000080:00200000:3.0:1789760728.040945:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23511040, bytes: 4096 00000080:00200000:3.0:1789760728.040953:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.040960:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.040962:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23511040, 23515136) 0 0 00000080:00200000:3.0:1789760728.040966:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.040979:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5740, 5740] 00020000:00200000:3.0:1789760728.040985:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23511040 chunk: [23511040, 23515136) 23515136, 4096 00020000:00200000:3.0:1789760728.040989:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.041003:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5740, 5740] 00000008:00010000:3.0:1789760728.041016:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.041021:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760728.041041:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23511040, 23515136) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.041053:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23511040=0x166c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.041095:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.041097:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23515136 00000020:00010000:3.0:1789760728.041104:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.041107:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.041109:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.041112:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.041115:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.041120:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.041126:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.041139:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760728.041215:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad180 - object 0x240000400:5 offset 23511040<>23515136 00000100:00100000:2.0:1789760728.041274:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512594176:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760728.081560:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973059 gen 3 00000008:00010000:2.0:1789760728.081598:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760728.081604:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760728.081614:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760728.081674:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:2.0:1789760728.081700:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512594176:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760728.081772:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5740, 5740] 00000080:00200000:3.0:1789760728.081785:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.081790:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.081803:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23515136 i_blocks 47111 00000020:00200000:3.0:1789760728.081806:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.081815:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.081841:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23515136, count: 4096 00000080:00200000:3.0:1789760728.081846:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23515136, bytes: 4096 00000080:00200000:3.0:1789760728.081853:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.081860:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.081862:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23515136, 23519232) 0 0 00000080:00200000:3.0:1789760728.081866:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.081879:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5741, 5741] 00020000:00200000:3.0:1789760728.081885:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23515136 chunk: [23515136, 23519232) 23519232, 4096 00020000:00200000:3.0:1789760728.081889:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.081984:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5741, 5741] 00000008:00010000:3.0:1789760728.082001:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.082005:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760728.082014:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23515136, 23519232) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.082024:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23515136=0x166d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.082119:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.082122:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23519232 00000020:00010000:3.0:1789760728.082128:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.082132:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.082135:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.082138:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.082140:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.082147:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.082152:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.082165:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760728.082235:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af100 - object 0x240000400:5 offset 23515136<>23519232 00000100:00100000:2.0:1789760728.082297:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512594304:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760728.138315:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973060 gen 3 00000008:00010000:2.0:1789760728.138353:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760728.138360:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760728.138369:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760728.138388:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760728.138412:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512594304:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760728.138538:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5741, 5741] 00000080:00200000:3.0:1789760728.138552:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.138557:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.138570:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23519232 i_blocks 47111 00000020:00200000:3.0:1789760728.138573:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.138582:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.138609:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23519232, count: 4096 00000080:00200000:3.0:1789760728.138615:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23519232, bytes: 4096 00000080:00200000:3.0:1789760728.138624:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.138631:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.138634:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23519232, 23523328) 0 0 00000080:00200000:3.0:1789760728.138638:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.138650:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5742, 5742] 00020000:00200000:3.0:1789760728.138657:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23519232 chunk: [23519232, 23523328) 23523328, 4096 00020000:00200000:3.0:1789760728.138660:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.138675:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5742, 5742] 00000008:00010000:3.0:1789760728.138690:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.138695:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760728.138706:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23519232, 23523328) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.138718:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23519232=0x166e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.138765:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.138769:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23523328 00000020:00010000:3.0:1789760728.138776:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.138780:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.138783:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.138800:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.138804:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.138811:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.138818:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.138832:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760728.138956:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aea00 - object 0x240000400:5 offset 23519232<>23523328 00000100:00100000:2.0:1789760728.139388:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512594432:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760728.181086:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973061 gen 3 00000008:00010000:2.0:1789760728.181124:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760728.181131:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760728.181140:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760728.181160:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760728.181183:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512594432:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760728.182440:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5742, 5742] 00000080:00200000:3.0:1789760728.182454:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.182460:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.182471:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23523328 i_blocks 47111 00000020:00200000:3.0:1789760728.182475:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.182483:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.182507:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23523328, count: 4096 00000080:00200000:3.0:1789760728.182512:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23523328, bytes: 4096 00000080:00200000:3.0:1789760728.182520:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.182539:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.182541:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23523328, 23527424) 0 0 00000080:00200000:3.0:1789760728.182545:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.182556:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5743, 5743] 00020000:00200000:3.0:1789760728.182563:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23523328 chunk: [23523328, 23527424) 23527424, 4096 00020000:00200000:3.0:1789760728.182566:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.182582:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5743, 5743] 00000008:00010000:3.0:1789760728.182595:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.182599:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760728.182608:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23523328, 23527424) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.182617:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23523328=0x166f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.182665:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.182668:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23527424 00000020:00010000:3.0:1789760728.182676:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.182679:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.182682:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.182686:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.182688:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.182695:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.182701:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.182716:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760728.182794:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac380 - object 0x240000400:5 offset 23523328<>23527424 00000100:00100000:2.0:1789760728.182871:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512594560:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760728.229845:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973062 gen 3 00000008:00010000:2.0:1789760728.229881:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760728.229888:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760728.229897:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760728.229948:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760728.229971:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512594560:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760728.230059:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5743, 5743] 00000080:00200000:3.0:1789760728.230074:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.230079:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.230090:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23527424 i_blocks 47111 00000020:00200000:3.0:1789760728.230093:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.230101:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.230127:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23527424, count: 4096 00000080:00200000:3.0:1789760728.230133:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23527424, bytes: 4096 00000080:00200000:3.0:1789760728.230142:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.230148:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.230150:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23527424, 23531520) 0 0 00000080:00200000:3.0:1789760728.230154:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.230168:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5744, 5744] 00020000:00200000:3.0:1789760728.230175:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23527424 chunk: [23527424, 23531520) 23531520, 4096 00020000:00200000:3.0:1789760728.230179:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.230204:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5744, 5744] 00000008:00010000:3.0:1789760728.230217:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.230222:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760728.230231:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23527424, 23531520) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.230242:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23527424=0x1670000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.230286:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.230289:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23531520 00000020:00010000:3.0:1789760728.230296:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.230299:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.230301:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.230304:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.230307:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.230313:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.230318:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.230331:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760728.230406:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac000 - object 0x240000400:5 offset 23527424<>23531520 00000100:00100000:2.0:1789760728.230627:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512594688:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760728.269395:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973063 gen 3 00000008:00010000:2.0:1789760728.269433:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760728.269439:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760728.269448:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760728.269467:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760728.269491:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512594688:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760728.269888:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5744, 5744] 00000080:00200000:3.0:1789760728.269933:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.269938:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.269951:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23531520 i_blocks 47111 00000020:00200000:3.0:1789760728.269954:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.269964:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.269991:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23531520, count: 4096 00000080:00200000:3.0:1789760728.269997:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23531520, bytes: 4096 00000080:00200000:3.0:1789760728.270005:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.270013:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.270014:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23531520, 23535616) 0 0 00000080:00200000:3.0:1789760728.270018:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.270031:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5745, 5745] 00020000:00200000:3.0:1789760728.270037:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23531520 chunk: [23531520, 23535616) 23535616, 4096 00020000:00200000:3.0:1789760728.270041:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.270057:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5745, 5745] 00000008:00010000:3.0:1789760728.270070:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.270081:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760728.270091:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23531520, 23535616) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.270101:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23531520=0x1671000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.270151:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.270154:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23535616 00000020:00010000:3.0:1789760728.270165:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.270168:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.270171:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.270176:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.270178:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.270184:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.270190:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.270204:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760728.270281:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae300 - object 0x240000400:5 offset 23531520<>23535616 00000100:00100000:2.0:1789760728.270342:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512594816:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760728.316457:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973064 gen 3 00000008:00010000:2.0:1789760728.316495:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760728.316501:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760728.316510:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760728.316542:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:2.0:1789760728.316564:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512594816:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760728.316668:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5745, 5745] 00000080:00200000:3.0:1789760728.316681:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.316686:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.316699:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23535616 i_blocks 47111 00000020:00200000:3.0:1789760728.316702:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.316711:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.316737:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23535616, count: 4096 00000080:00200000:3.0:1789760728.316744:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23535616, bytes: 4096 00000080:00200000:3.0:1789760728.316754:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.316762:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.316764:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23535616, 23539712) 0 0 00000080:00200000:3.0:1789760728.316767:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.316782:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5746, 5746] 00020000:00200000:3.0:1789760728.316789:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23535616 chunk: [23535616, 23539712) 23539712, 4096 00020000:00200000:3.0:1789760728.316792:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.316810:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5746, 5746] 00000008:00010000:3.0:1789760728.316825:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.316830:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760728.316839:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23535616, 23539712) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.316851:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23535616=0x1672000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.316897:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.316933:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23539712 00000020:00010000:3.0:1789760728.316940:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.316943:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.316947:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.316950:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.316953:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.316960:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.316965:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.316979:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760728.317056:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af480 - object 0x240000400:5 offset 23535616<>23539712 00000100:00100000:2.0:1789760728.317310:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512594944:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760728.365771:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760728.365783:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760728.365801:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760728.365820:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760728.365844:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512594944:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760728.365923:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5746, 5746] 00000080:00200000:3.0:1789760728.365935:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.365940:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.365951:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23539712 i_blocks 47111 00000020:00200000:3.0:1789760728.365954:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.365962:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.365988:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23539712, count: 4096 00000080:00200000:3.0:1789760728.366006:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23539712, bytes: 4096 00000080:00200000:3.0:1789760728.366014:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.366021:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.366023:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23539712, 23543808) 0 0 00000080:00200000:3.0:1789760728.366027:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.366040:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5747, 5747] 00020000:00200000:3.0:1789760728.366046:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23539712 chunk: [23539712, 23543808) 23543808, 4096 00020000:00200000:3.0:1789760728.366049:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.366065:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5747, 5747] 00000008:00010000:3.0:1789760728.366078:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.366082:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760728.366091:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23539712, 23543808) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.366101:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23539712=0x1673000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.366144:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.366147:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23543808 00000020:00010000:3.0:1789760728.366154:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.366157:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.366159:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.366175:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.366177:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.366183:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.366189:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.366202:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760728.366284:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac700 - object 0x240000400:5 offset 23539712<>23543808 00000100:00100000:2.0:1789760728.366448:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760728.366462:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512595072:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760728.399652:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973066 gen 3 00000008:00010000:2.0:1789760728.399696:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760728.399703:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760728.399711:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760728.399729:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760728.399751:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512595072:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760728.399881:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5747, 5747] 00000080:00200000:3.0:1789760728.399891:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.399895:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.399924:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23543808 i_blocks 47111 00000020:00200000:3.0:1789760728.399929:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.399937:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.399962:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23543808, count: 4096 00000080:00200000:3.0:1789760728.399969:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23543808, bytes: 4096 00000080:00200000:3.0:1789760728.399976:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.399981:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.399983:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23543808, 23547904) 0 0 00000080:00200000:3.0:1789760728.399986:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.399996:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5748, 5748] 00020000:00200000:3.0:1789760728.400001:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23543808 chunk: [23543808, 23547904) 23547904, 4096 00020000:00200000:3.0:1789760728.400004:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.400015:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5748, 5748] 00000008:00010000:3.0:1789760728.400025:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.400029:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760728.400036:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23543808, 23547904) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.400044:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23543808=0x1674000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.400078:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.400079:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23547904 00000020:00010000:3.0:1789760728.400084:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.400087:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.400088:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.400091:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.400092:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.400096:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.400100:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.400109:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760728.400173:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 23543808<>23547904 00000100:00100000:2.0:1789760728.400397:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512595200:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760728.430123:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973067 gen 3 00000008:00010000:2.0:1789760728.430164:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760728.430170:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760728.430181:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760728.430193:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760728.430216:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512595200:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760728.430343:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5748, 5748] 00000080:00200000:3.0:1789760728.430356:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.430361:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.430374:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23547904 i_blocks 47111 00000020:00200000:3.0:1789760728.430378:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.430389:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.430417:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23547904, count: 4096 00000080:00200000:3.0:1789760728.430426:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23547904, bytes: 4096 00000080:00200000:3.0:1789760728.430432:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.430438:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.430440:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23547904, 23552000) 0 0 00000080:00200000:3.0:1789760728.430444:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.430456:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5749, 5749] 00020000:00200000:3.0:1789760728.430462:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23547904 chunk: [23547904, 23552000) 23552000, 4096 00020000:00200000:3.0:1789760728.430466:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.430480:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5749, 5749] 00000008:00010000:3.0:1789760728.430493:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.430498:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760728.430505:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23547904, 23552000) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.430516:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23547904=0x1675000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.430692:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.430695:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23552000 00000020:00010000:3.0:1789760728.430703:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.430706:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.430709:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.430711:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.430714:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.430719:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.430724:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.430736:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760728.430872:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad500 - object 0x240000400:5 offset 23547904<>23552000 00000100:00100000:2.0:1789760728.430991:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760728.431005:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512595328:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760728.473111:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973068 gen 3 00000008:00010000:2.0:1789760728.473151:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760728.473157:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760728.473166:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760728.473224:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760728.473246:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512595328:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760728.473319:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5749, 5749] 00000080:00200000:3.0:1789760728.473332:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.473337:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.473348:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23552000 i_blocks 47111 00000020:00200000:3.0:1789760728.473351:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.473359:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.473384:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23552000, count: 4096 00000080:00200000:3.0:1789760728.473391:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23552000, bytes: 4096 00000080:00200000:3.0:1789760728.473399:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.473406:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.473408:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23552000, 23556096) 0 0 00000080:00200000:3.0:1789760728.473412:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.473425:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5750, 5750] 00020000:00200000:3.0:1789760728.473432:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23552000 chunk: [23552000, 23556096) 23556096, 4096 00020000:00200000:3.0:1789760728.473436:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.473451:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5750, 5750] 00000008:00010000:3.0:1789760728.473637:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.473643:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760728.473654:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23552000, 23556096) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.473665:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23552000=0x1676000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.473712:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.473715:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23556096 00000020:00010000:3.0:1789760728.473721:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.473725:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.473728:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.473731:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.473734:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.473740:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.473746:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.473761:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760728.473835:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107afb80 - object 0x240000400:5 offset 23552000<>23556096 00000100:00100000:2.0:1789760728.473892:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512595456:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760728.508870:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973069 gen 3 00000008:00010000:2.0:1789760728.508996:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760728.509003:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760728.509013:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760728.509032:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000080:00200000:3.0:1789760728.509130:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5750, 5750] 00000080:00200000:3.0:1789760728.509154:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.509159:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.509171:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23556096 i_blocks 47111 00000020:00200000:3.0:1789760728.509174:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.509182:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000100:00100000:2.0:1789760728.509279:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512595456:192.168.203.156@tcp:4:dd.0 00000080:00200008:3.0:1789760728.509393:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23556096, count: 4096 00000080:00200000:3.0:1789760728.509400:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23556096, bytes: 4096 00000080:00200000:3.0:1789760728.509409:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.509416:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.509418:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23556096, 23560192) 0 0 00000080:00200000:3.0:1789760728.509421:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.509434:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5751, 5751] 00020000:00200000:3.0:1789760728.509441:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23556096 chunk: [23556096, 23560192) 23560192, 4096 00020000:00200000:3.0:1789760728.509445:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.509651:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5751, 5751] 00000008:00010000:3.0:1789760728.509669:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.509674:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760728.509685:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23556096, 23560192) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.509696:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23556096=0x1677000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.509743:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.509746:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23560192 00000020:00010000:3.0:1789760728.509754:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.509757:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.509760:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.509763:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.509765:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.509772:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.509777:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.509792:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760728.509874:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 23556096<>23560192 00000100:00100000:2.0:1789760728.510014:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512595584:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760728.558813:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973070 gen 3 00000008:00010000:2.0:1789760728.558856:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760728.558862:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760728.558872:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760728.559635:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760728.559665:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512595584:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760728.560052:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5751, 5751] 00000080:00200000:3.0:1789760728.560077:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760728.560083:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760728.560102:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23560192 i_blocks 47111 00000020:00200000:3.0:1789760728.560108:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760728.560123:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760728.560161:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23560192, count: 4096 00000080:00200000:3.0:1789760728.560169:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23560192, bytes: 4096 00000080:00200000:3.0:1789760728.560183:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760728.560194:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760728.560196:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23560192, 23564288) 0 0 00000080:00200000:3.0:1789760728.560201:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760728.560219:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5752, 5752] 00020000:00200000:3.0:1789760728.560227:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23560192 chunk: [23560192, 23564288) 23564288, 4096 00020000:00200000:3.0:1789760728.560232:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760728.560253:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5752, 5752] 00000008:00010000:3.0:1789760728.560272:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760728.560278:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760728.560290:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23560192, 23564288) append 0 append lockpos 0 00000080:00200000:3.0:1789760728.560304:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23560192=0x1678000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760728.560377:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760728.560380:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23564288 00000020:00010000:3.0:1789760728.561771:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760728.561779:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.562362:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760728.562369:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760728.562372:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760728.562387:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760728.562402:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760728.562445:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760728.562597:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 23560192<>23564288 00000100:00100000:3.0:1789760728.562671:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512595712:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760728.610475:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973071 gen 3 00000008:00010000:2.0:1789760728.610515:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760728.610532:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760728.610543:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760728.610568:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:2.0:1789760728.610592:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512595712:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760728.612115:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760728.612135:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512595840:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760728.651939:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760728.651951:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760728.651961:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760728.651984:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760728.651999:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512595840:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760728.657976:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512595968:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760728.708462:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973073 gen 3 00000008:00010000:1.0:1789760728.708511:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760728.708528:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760728.708537:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760728.708562:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760728.708584:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512595968:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760728.711027:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760728.711047:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512596096:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760728.751536:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973074 gen 3 00000008:00010000:1.0:1789760728.751574:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760728.751687:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760728.751699:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760728.751709:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760728.751732:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512596096:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760728.753288:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512596224:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760728.796311:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973075 gen 3 00000008:00010000:1.0:1789760728.796351:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760728.796359:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760728.796371:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760728.796401:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760728.796436:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512596224:192.168.203.156@tcp:4:dd.0 00000020:00200000:2.0:1789760728.797796:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760728.797801:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23584768 00000020:00010000:2.0:1789760728.797809:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760728.797813:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760728.797816:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c173b0 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760728.797819:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760728.797822:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c173b0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760728.797829:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760728.797837:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760728.797852:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760728.798228:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 23580672<>23584768 00000100:00100000:3.0:1789760728.798300:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512596352:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760728.851054:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760728.851067:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760728.851077:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760728.851102:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760728.851128:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512596352:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760728.851827:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760728.851852:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512596480:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760728.861317:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e6cf7267-9b19-462a-a263-14115df15dab:2360:1876699512596608:192.168.203.156@tcp:400:kworker.0 00010000:00010000:2.0:1789760728.861400:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760728.861554:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2360:1876699512596864:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760728.861602:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760728.861608:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512596736:192.168.203.156@tcp:400:kworker.0 00000100:00100000:2.0:1789760728.863836:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e6cf7267-9b19-462a-a263-14115df15dab:2360:1876699512596608:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760728.865872:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512596736:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760728.865890:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2360:1876699512596864:192.168.203.156@tcp:400:kworker.0 00000100:00100000:1.0:1789760728.893068:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973077 gen 3 00000008:00010000:1.0:1789760728.893119:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760728.893125:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760728.893135:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760728.893148:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760728.893173:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512596480:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760728.893284:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5758, 5758] 00000080:00200000:2.0:1789760728.893298:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760728.893304:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760728.893316:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23588864 i_blocks 47111 00000020:00200000:2.0:1789760728.893319:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760728.893328:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760728.893361:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23588864, count: 4096 00000080:00200000:2.0:1789760728.893367:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23588864, bytes: 4096 00000080:00200000:2.0:1789760728.893376:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760728.893385:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760728.893388:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23588864, 23592960) 0 0 00000080:00200000:2.0:1789760728.893406:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760728.893420:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5759, 5759] 00020000:00200000:2.0:1789760728.893427:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23588864 chunk: [23588864, 23592960) 23592960, 4096 00020000:00200000:2.0:1789760728.893431:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760728.893449:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5759, 5759] 00000008:00010000:2.0:1789760728.893464:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760728.893480:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760728.893491:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23588864, 23592960) append 0 append lockpos 0 00000080:00200000:2.0:1789760728.893503:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23588864=0x167f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760728.893557:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760728.893560:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23592960 00000020:00010000:2.0:1789760728.893567:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760728.893570:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760728.893572:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760728.893575:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760728.893578:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760728.893596:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760728.893602:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760728.893616:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760728.893708:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 23588864<>23592960 00000100:00100000:3.0:1789760728.893997:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760728.894016:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512596992:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760728.929725:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973078 gen 3 00000008:00010000:2.0:1789760728.929765:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760728.929772:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760728.929779:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760728.929885:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760728.929953:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512596992:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760728.930630:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512597120:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760728.977122:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973079 gen 3 00000008:00010000:1.0:1789760728.977159:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760728.977164:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760728.977173:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760728.977197:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760728.977220:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512597120:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760728.978089:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760728.978106:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512597248:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760729.019501:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973080 gen 3 00000008:00010000:1.0:1789760729.019611:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760729.019717:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760729.019728:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760729.019750:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760729.019774:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512597248:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760729.022798:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512597376:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760729.061243:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973081 gen 3 00000008:00010000:1.0:1789760729.061282:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760729.061288:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760729.061298:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760729.061321:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760729.061348:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512597376:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760729.062832:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760729.062851:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512597504:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760729.102615:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760729.102626:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760729.102635:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760729.102658:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760729.102683:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512597504:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760729.103782:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512597632:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760729.141599:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973082 gen 3 00000100:00100000:1.0:1789760729.141629:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005148380 x1876699512597632/t4294973083(4294973083) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760745 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760729.141669:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760729.141675:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760729.141684:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760729.141760:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760729.141782:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512597632:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760729.142544:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512597760:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760729.173669:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973084 gen 3 00000008:00010000:1.0:1789760729.173715:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760729.173721:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760729.173729:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760729.173805:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760729.173827:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512597760:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760729.174605:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512597888:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760729.218192:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973085 gen 3 00000008:00010000:1.0:1789760729.218229:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760729.218236:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760729.218245:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760729.218270:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760729.218294:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512597888:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760729.219153:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5766, 5766] 00000080:00200000:2.0:1789760729.219167:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760729.219172:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760729.219183:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23621632 i_blocks 47111 00000020:00200000:2.0:1789760729.219187:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760729.219196:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760729.219221:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23621632, count: 4096 00000080:00200000:2.0:1789760729.219228:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23621632, bytes: 4096 00000080:00200000:2.0:1789760729.219237:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760729.219245:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760729.219247:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23621632, 23625728) 0 0 00000080:00200000:2.0:1789760729.219251:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760729.219264:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5767, 5767] 00020000:00200000:2.0:1789760729.219271:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23621632 chunk: [23621632, 23625728) 23625728, 4096 00020000:00200000:2.0:1789760729.219274:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760729.219290:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5767, 5767] 00000008:00010000:2.0:1789760729.219303:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760729.219308:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760729.219317:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23621632, 23625728) append 0 append lockpos 0 00000080:00200000:2.0:1789760729.219328:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23621632=0x1687000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760729.219375:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760729.219378:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23625728 00000020:00010000:2.0:1789760729.219384:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760729.219388:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.219391:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.219394:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760729.219397:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760729.219403:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760729.219409:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760729.219424:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760729.219506:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 23621632<>23625728 00000100:00100000:3.0:1789760729.220325:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512598016:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760729.277511:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973086 gen 3 00000008:00010000:2.0:1789760729.277556:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760729.277562:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760729.277570:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760729.277645:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760729.277670:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512598016:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760729.278721:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512598144:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760729.350339:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5768, 5768] 00000080:00200000:2.0:1789760729.350353:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760729.350359:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760729.350372:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23629824 i_blocks 47111 00000020:00200000:2.0:1789760729.350376:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760729.350384:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760729.350412:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23629824, count: 4096 00000080:00200000:2.0:1789760729.350419:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23629824, bytes: 4096 00000080:00200000:2.0:1789760729.350427:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760729.350434:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760729.350436:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23629824, 23633920) 0 0 00000080:00200000:2.0:1789760729.350440:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760729.350454:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5769, 5769] 00020000:00200000:2.0:1789760729.350462:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23629824 chunk: [23629824, 23633920) 23633920, 4096 00020000:00200000:2.0:1789760729.350466:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760729.350481:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5769, 5769] 00000008:00010000:2.0:1789760729.350497:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760729.350502:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760729.350512:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23629824, 23633920) append 0 append lockpos 0 00000080:00200000:2.0:1789760729.350534:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23629824=0x1689000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760729.350581:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760729.350584:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23633920 00000020:00010000:2.0:1789760729.350590:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760729.350595:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.350597:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.350601:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760729.350603:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760729.350610:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760729.350616:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760729.350630:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760729.350708:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 23629824<>23633920 00000100:00100000:2.0:1789760729.350748:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760729.350761:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512598272:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760729.390036:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973088 gen 3 00000008:00010000:2.0:1789760729.390074:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760729.390081:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760729.390091:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760729.390248:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760729.390274:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512598272:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760729.427127:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5770, 5770] 00000080:00200000:1.0:1789760729.427139:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760729.427144:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760729.427155:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23638016 i_blocks 47111 00000020:00200000:1.0:1789760729.427159:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760729.427167:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760729.427288:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23638016, count: 4096 00000080:00200000:1.0:1789760729.427295:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23638016, bytes: 4096 00000080:00200000:1.0:1789760729.427302:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760729.427309:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760729.427310:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23638016, 23642112) 0 0 00000080:00200000:1.0:1789760729.427314:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760729.427327:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5771, 5771] 00020000:00200000:1.0:1789760729.427333:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23638016 chunk: [23638016, 23642112) 23642112, 4096 00020000:00200000:1.0:1789760729.427336:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760729.427349:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5771, 5771] 00000008:00010000:1.0:1789760729.427361:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760729.427366:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760729.427374:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23638016, 23642112) append 0 append lockpos 0 00000080:00200000:1.0:1789760729.427384:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23638016=0x168b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760729.427422:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760729.427425:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23642112 00000020:00010000:1.0:1789760729.427430:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760729.427434:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.427436:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.427439:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760729.427442:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760729.427448:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760729.427452:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760729.427465:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:1.0:1789760729.427605:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 23638016<>23642112 00000100:00100000:1.0:1789760729.427637:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760729.427648:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512598528:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760729.461025:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973090 gen 3 00000008:00010000:1.0:1789760729.461059:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42240 00000020:00200000:1.0:1789760729.461065:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760729.461072:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760729.461097:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42240 00000100:00100000:1.0:1789760729.461119:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512598528:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760729.461695:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5771, 5771] 00000080:00200000:2.0:1789760729.461709:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760729.461713:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760729.461725:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23642112 i_blocks 47111 00000020:00200000:2.0:1789760729.461729:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760729.461737:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760729.461763:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23642112, count: 4096 00000080:00200000:2.0:1789760729.461768:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23642112, bytes: 4096 00000080:00200000:2.0:1789760729.461777:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760729.461784:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760729.461786:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23642112, 23646208) 0 0 00000080:00200000:2.0:1789760729.461790:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760729.461802:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5772, 5772] 00020000:00200000:2.0:1789760729.461809:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23642112 chunk: [23642112, 23646208) 23646208, 4096 00020000:00200000:2.0:1789760729.461813:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760729.461828:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5772, 5772] 00000008:00010000:2.0:1789760729.461843:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760729.461848:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760729.461857:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23642112, 23646208) append 0 append lockpos 0 00000080:00200000:2.0:1789760729.461868:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23642112=0x168c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760729.461950:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760729.461953:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23646208 00000020:00010000:2.0:1789760729.461959:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760729.461962:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.461965:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.461968:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760729.461971:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760729.461977:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760729.461983:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760729.461998:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760729.462072:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 23642112<>23646208 00000100:00100000:3.0:1789760729.462283:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512598656:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760729.498177:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760729.498188:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760729.498197:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760729.498221:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760729.498246:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512598656:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760729.498388:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5772, 5772] 00000080:00200000:1.0:1789760729.498402:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760729.498408:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760729.498420:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23646208 i_blocks 47111 00000020:00200000:1.0:1789760729.498423:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760729.498431:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760729.498459:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23646208, count: 4096 00000080:00200000:1.0:1789760729.498466:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23646208, bytes: 4096 00000080:00200000:1.0:1789760729.498473:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760729.498480:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760729.498482:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23646208, 23650304) 0 0 00000080:00200000:1.0:1789760729.498486:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760729.498500:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5773, 5773] 00020000:00200000:1.0:1789760729.498506:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23646208 chunk: [23646208, 23650304) 23650304, 4096 00020000:00200000:1.0:1789760729.498510:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760729.498534:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5773, 5773] 00000008:00010000:1.0:1789760729.498555:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760729.498560:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760729.498570:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23646208, 23650304) append 0 append lockpos 0 00000080:00200000:1.0:1789760729.498579:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23646208=0x168d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760729.498623:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760729.498627:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23650304 00000020:00010000:1.0:1789760729.498633:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760729.498636:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.498639:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.498642:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760729.498646:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760729.498652:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760729.498658:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760729.498672:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42240 00000008:00100000:1.0:1789760729.498750:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 23646208<>23650304 00000080:00200000:1.0:1789760729.535145:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5773, 5773] 00000080:00200000:1.0:1789760729.535157:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760729.535161:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760729.535171:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23650304 i_blocks 47111 00000020:00200000:1.0:1789760729.535174:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760729.535189:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760729.535214:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23650304, count: 4096 00000080:00200000:1.0:1789760729.535219:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23650304, bytes: 4096 00000080:00200000:1.0:1789760729.535226:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760729.535233:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760729.535235:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23650304, 23654400) 0 0 00000080:00200000:1.0:1789760729.535239:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760729.535250:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5774, 5774] 00020000:00200000:1.0:1789760729.535256:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23650304 chunk: [23650304, 23654400) 23654400, 4096 00020000:00200000:1.0:1789760729.535260:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760729.535274:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5774, 5774] 00000008:00010000:1.0:1789760729.535285:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760729.535290:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760729.535298:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23650304, 23654400) append 0 append lockpos 0 00000080:00200000:1.0:1789760729.535307:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23650304=0x168e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760729.535345:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760729.535348:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23654400 00000020:00010000:1.0:1789760729.535353:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760729.535357:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.535359:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.535362:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760729.535365:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760729.535370:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760729.535375:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760729.535386:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f429c0 00000008:00100000:1.0:1789760729.535453:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95c00 - object 0x240000400:5 offset 23650304<>23654400 00000080:00200000:1.0:1789760729.572055:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5774, 5774] 00000080:00200000:1.0:1789760729.572066:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760729.572071:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760729.572082:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23654400 i_blocks 47111 00000020:00200000:1.0:1789760729.572086:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760729.572093:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760729.572116:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23654400, count: 4096 00000080:00200000:1.0:1789760729.572121:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23654400, bytes: 4096 00000080:00200000:1.0:1789760729.572128:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760729.572135:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760729.572137:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23654400, 23658496) 0 0 00000080:00200000:1.0:1789760729.572141:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760729.572153:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5775, 5775] 00020000:00200000:1.0:1789760729.572159:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23654400 chunk: [23654400, 23658496) 23658496, 4096 00020000:00200000:1.0:1789760729.572163:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760729.572176:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5775, 5775] 00000008:00010000:1.0:1789760729.572198:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760729.572202:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760729.572210:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23654400, 23658496) append 0 append lockpos 0 00000080:00200000:1.0:1789760729.572220:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23654400=0x168f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760729.572256:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760729.572258:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23658496 00000020:00010000:1.0:1789760729.572264:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760729.572267:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.572270:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.572273:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760729.572276:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760729.572281:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760729.572286:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760729.572297:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760729.572365:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 23654400<>23658496 00000100:00100000:1.0:1789760729.572400:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760729.572412:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512599040:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760729.607815:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973094 gen 3 00000008:00010000:1.0:1789760729.607863:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760729.607869:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760729.607877:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760729.608132:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760729.608157:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512599040:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760729.608702:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5775, 5775] 00000080:00200000:2.0:1789760729.608716:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760729.608721:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760729.608732:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23658496 i_blocks 47111 00000020:00200000:2.0:1789760729.608736:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760729.608744:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760729.608771:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23658496, count: 4096 00000080:00200000:2.0:1789760729.608777:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23658496, bytes: 4096 00000080:00200000:2.0:1789760729.608786:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760729.608802:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760729.608805:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23658496, 23662592) 0 0 00000080:00200000:2.0:1789760729.608808:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760729.609041:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5776, 5776] 00020000:00200000:2.0:1789760729.609049:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23658496 chunk: [23658496, 23662592) 23662592, 4096 00020000:00200000:2.0:1789760729.609054:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760729.609069:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5776, 5776] 00000008:00010000:2.0:1789760729.609082:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760729.609087:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760729.609096:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23658496, 23662592) append 0 append lockpos 0 00000080:00200000:2.0:1789760729.609105:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23658496=0x1690000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760729.609150:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760729.609152:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23662592 00000020:00010000:2.0:1789760729.609159:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760729.609162:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.609165:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.609167:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760729.609170:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760729.609175:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760729.609181:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760729.609210:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760729.609289:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 23658496<>23662592 00000100:00100000:3.0:1789760729.609362:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512599168:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760729.647267:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760729.647278:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760729.647287:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760729.647307:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760729.647331:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512599168:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760729.647470:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5776, 5776] 00000080:00200000:1.0:1789760729.647482:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760729.647487:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760729.647498:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23662592 i_blocks 47111 00000020:00200000:1.0:1789760729.647501:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760729.647509:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760729.647542:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23662592, count: 4096 00000080:00200000:1.0:1789760729.647548:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23662592, bytes: 4096 00000080:00200000:1.0:1789760729.647556:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760729.647564:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760729.647566:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23662592, 23666688) 0 0 00000080:00200000:1.0:1789760729.647570:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760729.647582:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5777, 5777] 00020000:00200000:1.0:1789760729.647589:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23662592 chunk: [23662592, 23666688) 23666688, 4096 00020000:00200000:1.0:1789760729.647593:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760729.647607:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5777, 5777] 00000008:00010000:1.0:1789760729.647619:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760729.647624:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760729.647633:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23662592, 23666688) append 0 append lockpos 0 00000080:00200000:1.0:1789760729.647643:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23662592=0x1691000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760729.647686:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760729.647689:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23666688 00000020:00010000:1.0:1789760729.647695:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760729.647699:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.647701:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.647705:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760729.647707:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760729.647713:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760729.647718:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760729.647730:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760729.647806:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 23662592<>23666688 00000080:00200000:1.0:1789760729.678339:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5777, 5777] 00000080:00200000:1.0:1789760729.678351:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760729.678356:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760729.678367:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23666688 i_blocks 47111 00000020:00200000:1.0:1789760729.678370:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760729.678377:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760729.678404:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23666688, count: 4096 00000080:00200000:1.0:1789760729.678409:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23666688, bytes: 4096 00000080:00200000:1.0:1789760729.678416:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760729.678422:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760729.678425:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23666688, 23670784) 0 0 00000080:00200000:1.0:1789760729.678428:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760729.678440:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5778, 5778] 00020000:00200000:1.0:1789760729.678447:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23666688 chunk: [23666688, 23670784) 23670784, 4096 00020000:00200000:1.0:1789760729.678450:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760729.678464:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5778, 5778] 00000008:00010000:1.0:1789760729.678477:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760729.678482:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760729.678490:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23666688, 23670784) append 0 append lockpos 0 00000080:00200000:1.0:1789760729.678500:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23666688=0x1692000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760729.678552:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760729.678554:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23670784 00000020:00010000:1.0:1789760729.678561:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760729.678564:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.678567:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.678570:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760729.678573:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760729.678579:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760729.678585:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760729.678597:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760729.678665:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 23666688<>23670784 00000100:00100000:1.0:1789760729.711316:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973097 gen 3 00000008:00010000:1.0:1789760729.711359:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760729.711366:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760729.711373:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760729.711394:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760729.711414:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512599424:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760729.711498:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5778, 5778] 00000080:00200000:2.0:1789760729.711518:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760729.711523:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760729.711535:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23670784 i_blocks 47111 00000020:00200000:2.0:1789760729.711538:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760729.711546:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760729.711571:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23670784, count: 4096 00000080:00200000:2.0:1789760729.711578:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23670784, bytes: 4096 00000080:00200000:2.0:1789760729.711586:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760729.711592:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760729.711594:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23670784, 23674880) 0 0 00000080:00200000:2.0:1789760729.711598:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760729.711612:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5779, 5779] 00020000:00200000:2.0:1789760729.711618:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23670784 chunk: [23670784, 23674880) 23674880, 4096 00020000:00200000:2.0:1789760729.711621:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760729.711635:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5779, 5779] 00000008:00010000:2.0:1789760729.711648:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760729.711653:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760729.711662:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23670784, 23674880) append 0 append lockpos 0 00000080:00200000:2.0:1789760729.711672:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23670784=0x1693000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760729.711715:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760729.711717:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23674880 00000020:00010000:2.0:1789760729.711724:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760729.711727:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.711730:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.711733:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760729.711735:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760729.711742:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760729.711747:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760729.711761:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760729.711850:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 23670784<>23674880 00000100:00100000:2.0:1789760729.711889:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760729.711947:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512599552:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760729.746095:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973098 gen 3 00000008:00010000:2.0:1789760729.746131:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760729.746138:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760729.746147:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760729.746171:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760729.746195:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512599552:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760729.746294:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5779, 5779] 00000080:00200000:1.0:1789760729.746306:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760729.746312:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760729.746326:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23674880 i_blocks 47111 00000020:00200000:1.0:1789760729.746330:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760729.746340:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760729.746367:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23674880, count: 4096 00000080:00200000:1.0:1789760729.746374:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23674880, bytes: 4096 00000080:00200000:1.0:1789760729.746382:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760729.746391:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760729.746393:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23674880, 23678976) 0 0 00000080:00200000:1.0:1789760729.746396:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760729.746410:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5780, 5780] 00020000:00200000:1.0:1789760729.746418:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23674880 chunk: [23674880, 23678976) 23678976, 4096 00020000:00200000:1.0:1789760729.746422:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760729.746438:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5780, 5780] 00000008:00010000:1.0:1789760729.746452:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760729.746457:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760729.746466:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23674880, 23678976) append 0 append lockpos 0 00000080:00200000:1.0:1789760729.746475:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23674880=0x1694000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760729.746539:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760729.746542:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23678976 00000020:00010000:1.0:1789760729.746549:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760729.746552:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.746555:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.746558:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760729.746561:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760729.746567:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760729.746572:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760729.746586:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760729.746668:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 23674880<>23678976 00000100:00100000:1.0:1789760729.746707:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760729.746720:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512599680:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760729.778009:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973099 gen 3 00000008:00010000:1.0:1789760729.778046:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760729.778052:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760729.778060:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760729.778332:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760729.778357:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512599680:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760729.778434:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5780, 5780] 00000080:00200000:2.0:1789760729.778446:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760729.778452:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760729.778464:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23678976 i_blocks 47111 00000020:00200000:2.0:1789760729.778467:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760729.778476:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760729.778502:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23678976, count: 4096 00000080:00200000:2.0:1789760729.778508:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23678976, bytes: 4096 00000080:00200000:2.0:1789760729.778832:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760729.778839:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760729.778841:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23678976, 23683072) 0 0 00000080:00200000:2.0:1789760729.778845:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760729.778859:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5781, 5781] 00020000:00200000:2.0:1789760729.778865:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23678976 chunk: [23678976, 23683072) 23683072, 4096 00020000:00200000:2.0:1789760729.778869:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760729.778886:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5781, 5781] 00000008:00010000:2.0:1789760729.778934:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760729.778939:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760729.778950:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23678976, 23683072) append 0 append lockpos 0 00000080:00200000:2.0:1789760729.778960:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23678976=0x1695000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760729.779006:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760729.779008:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23683072 00000020:00010000:2.0:1789760729.779014:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760729.779017:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.779020:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.779023:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760729.779025:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760729.779031:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760729.779037:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760729.779052:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760729.779335:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 23678976<>23683072 00000100:00100000:3.0:1789760729.779415:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512599808:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760729.823480:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973100 gen 3 00000008:00010000:2.0:1789760729.823526:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760729.823532:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760729.823540:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760729.823622:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760729.823647:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512599808:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760729.825992:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5781, 5781] 00000080:00200000:1.0:1789760729.826018:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760729.826024:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760729.826038:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23683072 i_blocks 47111 00000020:00200000:1.0:1789760729.826046:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760729.826058:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760729.826085:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23683072, count: 4096 00000080:00200000:1.0:1789760729.826091:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23683072, bytes: 4096 00000080:00200000:1.0:1789760729.826102:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760729.826109:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760729.826111:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23683072, 23687168) 0 0 00000080:00200000:1.0:1789760729.826115:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760729.826128:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5782, 5782] 00020000:00200000:1.0:1789760729.826135:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23683072 chunk: [23683072, 23687168) 23687168, 4096 00020000:00200000:1.0:1789760729.826139:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760729.826155:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5782, 5782] 00000008:00010000:1.0:1789760729.826298:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760729.826303:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760729.826313:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23683072, 23687168) append 0 append lockpos 0 00000080:00200000:1.0:1789760729.826326:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23683072=0x1696000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760729.826389:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760729.826392:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23687168 00000020:00010000:1.0:1789760729.826398:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760729.826401:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.826405:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760729.826408:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760729.826411:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760729.826417:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760729.826423:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760729.826445:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760729.826532:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 23683072<>23687168 00000080:00200000:2.0:1789760729.865133:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5782, 5782] 00000080:00200000:2.0:1789760729.865146:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760729.865151:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760729.865170:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23687168 i_blocks 47111 00000020:00200000:2.0:1789760729.865173:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760729.865181:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760729.865207:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23687168, count: 4096 00000080:00200000:2.0:1789760729.865213:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23687168, bytes: 4096 00000080:00200000:2.0:1789760729.865220:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760729.865226:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760729.865229:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23687168, 23691264) 0 0 00000080:00200000:2.0:1789760729.865232:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760729.865245:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5783, 5783] 00020000:00200000:2.0:1789760729.865251:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23687168 chunk: [23687168, 23691264) 23691264, 4096 00020000:00200000:2.0:1789760729.865255:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760729.865271:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5783, 5783] 00000008:00010000:2.0:1789760729.865284:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760729.865290:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760729.865299:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23687168, 23691264) append 0 append lockpos 0 00000080:00200000:2.0:1789760729.865310:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23687168=0x1697000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760729.865353:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760729.865355:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23691264 00000020:00010000:2.0:1789760729.865362:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760729.865365:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.865367:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.865370:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760729.865373:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760729.865379:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760729.865384:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760729.865397:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760729.865470:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 23687168<>23691264 00000100:00100000:2.0:1789760729.865508:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760729.865531:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512600064:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760729.905718:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973102 gen 3 00000008:00010000:2.0:1789760729.905778:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760729.905988:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760729.905998:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760729.906025:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760729.906050:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512600064:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760729.968562:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5784, 5784] 00000080:00200000:2.0:1789760729.968577:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760729.968582:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760729.968594:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23695360 i_blocks 47111 00000020:00200000:2.0:1789760729.968598:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760729.968607:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760729.968631:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23695360, count: 4096 00000080:00200000:2.0:1789760729.968637:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23695360, bytes: 4096 00000080:00200000:2.0:1789760729.968647:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760729.968653:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760729.968732:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23695360, 23699456) 0 0 00000080:00200000:2.0:1789760729.968736:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760729.968750:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5785, 5785] 00020000:00200000:2.0:1789760729.968757:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23695360 chunk: [23695360, 23699456) 23699456, 4096 00020000:00200000:2.0:1789760729.968761:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760729.968778:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5785, 5785] 00000008:00010000:2.0:1789760729.968952:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760729.968958:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760729.968968:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23695360, 23699456) append 0 append lockpos 0 00000080:00200000:2.0:1789760729.968978:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23695360=0x1699000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760729.969098:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760729.969101:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23699456 00000020:00010000:2.0:1789760729.969108:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760729.969111:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.969114:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c2d8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760729.969117:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760729.969119:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760729.969126:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760729.969132:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760729.969153:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760729.969299:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 23695360<>23699456 00000100:00100000:3.0:1789760729.969526:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760729.969547:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512600320:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760730.008558:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760730.008571:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760730.008579:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760730.008599:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760730.008622:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512600320:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.011946:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512600448:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.053363:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973105 gen 3 00000008:00010000:1.0:1789760730.053409:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760730.053415:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.053422:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.053444:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760730.053466:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512600448:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.054478:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760730.054494:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512600576:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.112603:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973106 gen 3 00000008:00010000:1.0:1789760730.112640:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760730.112646:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.112655:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.112984:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760730.113008:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512600576:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.113415:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512600704:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.162166:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973107 gen 3 00000008:00010000:1.0:1789760730.162264:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760730.162270:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.162280:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.162299:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760730.162321:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512600704:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.164018:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512600832:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.209052:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973108 gen 3 00000008:00010000:1.0:1789760730.209086:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760730.209092:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.209102:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.209123:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760730.209145:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512600832:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.210249:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760730.210265:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512600960:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.266573:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973109 gen 3 00000008:00010000:1.0:1789760730.266611:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760730.266617:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.266625:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.266694:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760730.266721:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512600960:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.267407:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512601088:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.305762:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973110 gen 3 00000008:00010000:1.0:1789760730.305797:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760730.305803:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.305811:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.305881:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760730.305939:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512601088:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.308551:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512601216:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760730.351154:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760730.351165:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.351174:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.351183:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760730.351206:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512601216:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.352070:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760730.352085:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512601344:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.443021:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973111 gen 3 00000100:00100000:1.0:1789760730.443053:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00514aa00 x1876699512601344/t4294973112(4294973112) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760746 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760730.443089:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760730.443094:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.443102:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.443175:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760730.443196:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512601344:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.444896:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512601472:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.512869:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973113 gen 3 00000008:00010000:1.0:1789760730.512946:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760730.512952:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.512961:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.513036:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760730.513058:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512601472:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.516945:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512601600:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760730.559474:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760730.559487:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.559498:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.559628:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760730.559658:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512601600:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.561361:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512601728:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.596116:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973114 gen 3 00000100:00100000:1.0:1789760730.596148:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005148a80 x1876699512601728/t4294973115(4294973115) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760746 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760730.596200:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760730.596206:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.596214:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.596232:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760730.596252:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512601728:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.596993:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512601856:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.631112:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973115 gen 3 00000100:00100000:1.0:1789760730.631145:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00514a680 x1876699512601856/t4294973116(4294973116) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760746 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760730.631196:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760730.631203:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.631212:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.631343:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760730.631364:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512601856:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.634185:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760730.634205:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512601984:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.686710:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973117 gen 3 00000008:00010000:1.0:1789760730.686761:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760730.686768:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.686778:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.686875:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760730.686925:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512601984:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.688015:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512602112:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.752604:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973118 gen 3 00000008:00010000:1.0:1789760730.752642:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760730.752648:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.752657:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.752679:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760730.752704:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512602112:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.753236:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512602240:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.801213:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973119 gen 3 00000008:00010000:1.0:1789760730.801250:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760730.801257:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.801266:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.801289:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760730.801312:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512602240:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.801942:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512602368:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760730.866652:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973120 gen 3 00000008:00010000:1.0:1789760730.866694:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760730.866700:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760730.866710:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760730.866798:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760730.866822:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512602368:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760730.867699:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23764992 i_blocks 47111 00000020:00200000:1.0:1789760730.867704:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760730.867714:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760730.867854:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23764992, count: 4096 00000080:00200000:1.0:1789760730.867861:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23764992, bytes: 4096 00000080:00200000:1.0:1789760730.867870:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760730.867879:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760730.867881:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23764992, 23769088) 0 0 00000080:00200000:1.0:1789760730.867884:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760730.867982:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5802, 5802] 00020000:00200000:1.0:1789760730.867990:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23764992 chunk: [23764992, 23769088) 23769088, 4096 00020000:00200000:1.0:1789760730.867995:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760730.868012:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5802, 5802] 00000008:00010000:1.0:1789760730.868028:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760730.868034:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760730.868043:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23764992, 23769088) append 0 append lockpos 0 00000080:00200000:1.0:1789760730.868053:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23764992=0x16aa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760730.868100:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760730.868103:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23769088 00000020:00010000:1.0:1789760730.868109:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760730.868112:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760730.868115:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760730.868118:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760730.868121:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760730.868128:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760730.868134:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760730.868305:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f429c0 00000008:00100000:1.0:1789760730.868388:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 23764992<>23769088 00000080:00200000:3.0:1789760730.912191:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5802, 5802] 00000080:00200000:3.0:1789760730.912204:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760730.912208:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760730.912222:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23769088 i_blocks 47111 00000020:00200000:3.0:1789760730.912225:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760730.912233:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760730.912257:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23769088, count: 4096 00000080:00200000:3.0:1789760730.912263:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23769088, bytes: 4096 00000080:00200000:3.0:1789760730.912271:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760730.912278:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760730.912280:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23769088, 23773184) 0 0 00000080:00200000:3.0:1789760730.912284:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760730.912298:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5803, 5803] 00020000:00200000:3.0:1789760730.912306:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23769088 chunk: [23769088, 23773184) 23773184, 4096 00020000:00200000:3.0:1789760730.912310:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760730.912325:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5803, 5803] 00000008:00010000:3.0:1789760730.912340:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760730.912345:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760730.912354:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23769088, 23773184) append 0 append lockpos 0 00000080:00200000:3.0:1789760730.912365:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23769088=0x16ab000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760730.912411:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760730.912413:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23773184 00000020:00010000:3.0:1789760730.912420:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760730.912423:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760730.912426:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760730.912429:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760730.912432:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760730.912437:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760730.912443:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760730.912457:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760730.912541:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 23769088<>23773184 00000100:00100000:3.0:1789760730.912577:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760730.912589:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512602624:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760730.947297:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973122 gen 3 00000008:00010000:2.0:1789760730.947332:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760730.947339:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760730.947349:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760730.947373:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760730.947396:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512602624:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760730.947493:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5803, 5803] 00000080:00200000:3.0:1789760730.947519:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760730.947524:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760730.947536:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23773184 i_blocks 47111 00000020:00200000:3.0:1789760730.947539:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760730.947546:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760730.947570:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23773184, count: 4096 00000080:00200000:3.0:1789760730.947577:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23773184, bytes: 4096 00000080:00200000:3.0:1789760730.947586:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760730.947593:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760730.947595:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23773184, 23777280) 0 0 00000080:00200000:3.0:1789760730.947599:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760730.947611:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5804, 5804] 00020000:00200000:3.0:1789760730.947617:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23773184 chunk: [23773184, 23777280) 23777280, 4096 00020000:00200000:3.0:1789760730.947620:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760730.947636:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5804, 5804] 00000008:00010000:3.0:1789760730.947651:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760730.947656:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760730.947665:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23773184, 23777280) append 0 append lockpos 0 00000080:00200000:3.0:1789760730.947676:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23773184=0x16ac000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760730.947724:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760730.947728:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23777280 00000020:00010000:3.0:1789760730.947735:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760730.947738:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760730.947740:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760730.947743:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760730.947746:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760730.947753:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760730.947758:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760730.947774:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760730.947852:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad500 - object 0x240000400:5 offset 23773184<>23777280 00000100:00100000:2.0:1789760730.948184:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760730.948200:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512602752:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760730.987603:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973123 gen 3 00000008:00010000:2.0:1789760730.987637:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760730.987643:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760730.987653:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760730.987716:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760730.987740:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512602752:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760730.989138:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5804, 5804] 00000080:00200000:3.0:1789760730.989153:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760730.989159:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760730.989171:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23777280 i_blocks 47111 00000020:00200000:3.0:1789760730.989175:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760730.989184:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760730.989209:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23777280, count: 4096 00000080:00200000:3.0:1789760730.989216:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23777280, bytes: 4096 00000080:00200000:3.0:1789760730.989225:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760730.989233:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760730.989235:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23777280, 23781376) 0 0 00000080:00200000:3.0:1789760730.989238:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760730.989251:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5805, 5805] 00020000:00200000:3.0:1789760730.989257:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23777280 chunk: [23777280, 23781376) 23781376, 4096 00020000:00200000:3.0:1789760730.989325:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760730.989343:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5805, 5805] 00000008:00010000:3.0:1789760730.989357:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760730.989362:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760730.989371:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23777280, 23781376) append 0 append lockpos 0 00000080:00200000:3.0:1789760730.989381:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23777280=0x16ad000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760730.989428:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760730.989430:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23781376 00000020:00010000:3.0:1789760730.989436:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760730.989440:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760730.989443:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760730.989446:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760730.989449:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760730.989456:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760730.989461:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760730.989480:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760730.989571:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac700 - object 0x240000400:5 offset 23777280<>23781376 00000100:00100000:2.0:1789760730.989719:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512602880:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760731.039995:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760731.040007:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760731.040017:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760731.040035:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760731.040059:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512602880:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760731.040138:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5805, 5805] 00000080:00200000:3.0:1789760731.040150:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760731.040155:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760731.040166:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23781376 i_blocks 47111 00000020:00200000:3.0:1789760731.040169:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760731.040187:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760731.040213:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23781376, count: 4096 00000080:00200000:3.0:1789760731.040219:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23781376, bytes: 4096 00000080:00200000:3.0:1789760731.040226:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760731.040233:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760731.040235:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23781376, 23785472) 0 0 00000080:00200000:3.0:1789760731.040239:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760731.040252:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5806, 5806] 00020000:00200000:3.0:1789760731.040259:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23781376 chunk: [23781376, 23785472) 23785472, 4096 00020000:00200000:3.0:1789760731.040263:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760731.040276:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5806, 5806] 00000008:00010000:3.0:1789760731.040290:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760731.040295:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760731.040305:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23781376, 23785472) append 0 append lockpos 0 00000080:00200000:3.0:1789760731.040316:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23781376=0x16ae000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760731.040485:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760731.040488:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23785472 00000020:00010000:3.0:1789760731.040495:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760731.040498:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.040513:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.040516:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760731.040518:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760731.040524:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760731.040529:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760731.040543:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760731.040617:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af480 - object 0x240000400:5 offset 23781376<>23785472 00000100:00100000:2.0:1789760731.040677:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512603008:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760731.075962:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973125 gen 3 00000008:00010000:2.0:1789760731.076011:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760731.076017:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760731.076027:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760731.076040:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760731.076062:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512603008:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760731.076360:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5806, 5806] 00000080:00200000:3.0:1789760731.076372:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760731.076378:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760731.076390:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23785472 i_blocks 47111 00000020:00200000:3.0:1789760731.076394:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760731.076402:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760731.076426:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23785472, count: 4096 00000080:00200000:3.0:1789760731.076432:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23785472, bytes: 4096 00000080:00200000:3.0:1789760731.076440:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760731.076448:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760731.076450:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23785472, 23789568) 0 0 00000080:00200000:3.0:1789760731.076454:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760731.076467:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5807, 5807] 00020000:00200000:3.0:1789760731.076473:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23785472 chunk: [23785472, 23789568) 23789568, 4096 00020000:00200000:3.0:1789760731.076477:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760731.076491:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5807, 5807] 00000008:00010000:3.0:1789760731.076514:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760731.076520:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760731.076529:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23785472, 23789568) append 0 append lockpos 0 00000080:00200000:3.0:1789760731.076540:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23785472=0x16af000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760731.076580:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760731.076582:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23789568 00000020:00010000:3.0:1789760731.076589:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760731.076592:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.076595:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.076598:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760731.076601:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760731.076606:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760731.076612:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760731.076626:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760731.076701:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae300 - object 0x240000400:5 offset 23785472<>23789568 00000100:00100000:2.0:1789760731.076941:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760731.076958:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512603136:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760731.118343:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973126 gen 3 00000008:00010000:2.0:1789760731.118380:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760731.118386:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760731.118395:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760731.118411:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:2.0:1789760731.118434:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512603136:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760731.118592:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5807, 5807] 00000080:00200000:3.0:1789760731.118604:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760731.118610:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760731.118621:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23789568 i_blocks 47111 00000020:00200000:3.0:1789760731.118624:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760731.118632:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760731.118658:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23789568, count: 4096 00000080:00200000:3.0:1789760731.118664:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23789568, bytes: 4096 00000080:00200000:3.0:1789760731.118671:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760731.118678:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760731.118680:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23789568, 23793664) 0 0 00000080:00200000:3.0:1789760731.118684:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760731.118697:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5808, 5808] 00020000:00200000:3.0:1789760731.118705:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23789568 chunk: [23789568, 23793664) 23793664, 4096 00020000:00200000:3.0:1789760731.118709:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760731.118725:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5808, 5808] 00000008:00010000:3.0:1789760731.118739:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760731.118744:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760731.118754:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23789568, 23793664) append 0 append lockpos 0 00000080:00200000:3.0:1789760731.118766:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23789568=0x16b0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760731.118813:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760731.118816:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23793664 00000020:00010000:3.0:1789760731.118822:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760731.118826:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.118829:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.118832:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760731.118834:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760731.118841:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760731.118846:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760731.118860:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760731.119123:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac000 - object 0x240000400:5 offset 23789568<>23793664 00000100:00100000:2.0:1789760731.119427:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512603264:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760731.157691:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973127 gen 3 00000008:00010000:2.0:1789760731.157728:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760731.157733:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760731.157742:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760731.157763:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000080:00200000:3.0:1789760731.157988:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5808, 5808] 00000100:00100000:2.0:1789760731.157996:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512603264:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760731.158002:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760731.158006:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760731.158017:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23793664 i_blocks 47111 00000020:00200000:3.0:1789760731.158020:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760731.158027:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760731.158052:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23793664, count: 4096 00000080:00200000:3.0:1789760731.158057:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23793664, bytes: 4096 00000080:00200000:3.0:1789760731.158067:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760731.158073:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760731.158074:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23793664, 23797760) 0 0 00000080:00200000:3.0:1789760731.158077:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760731.158087:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5809, 5809] 00020000:00200000:3.0:1789760731.158092:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23793664 chunk: [23793664, 23797760) 23797760, 4096 00020000:00200000:3.0:1789760731.158094:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760731.158106:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5809, 5809] 00000008:00010000:3.0:1789760731.158115:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760731.158119:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760731.158126:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23793664, 23797760) append 0 append lockpos 0 00000080:00200000:3.0:1789760731.158134:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23793664=0x16b1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760731.158168:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760731.158170:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23797760 00000020:00010000:3.0:1789760731.158176:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760731.158179:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.158182:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.158184:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760731.158187:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760731.158192:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760731.158197:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760731.158211:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760731.158283:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac380 - object 0x240000400:5 offset 23793664<>23797760 00000100:00100000:2.0:1789760731.158465:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760731.158476:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512603392:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760731.207031:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973128 gen 3 00000008:00010000:2.0:1789760731.207070:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760731.207076:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760731.207085:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760731.207104:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760731.207129:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512603392:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760731.207273:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5809, 5809] 00000080:00200000:3.0:1789760731.207287:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760731.207292:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760731.207303:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23797760 i_blocks 47111 00000020:00200000:3.0:1789760731.207307:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760731.207314:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760731.207340:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23797760, count: 4096 00000080:00200000:3.0:1789760731.207346:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23797760, bytes: 4096 00000080:00200000:3.0:1789760731.207352:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760731.207360:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760731.207362:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23797760, 23801856) 0 0 00000080:00200000:3.0:1789760731.207365:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760731.207378:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5810, 5810] 00020000:00200000:3.0:1789760731.207385:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23797760 chunk: [23797760, 23801856) 23801856, 4096 00020000:00200000:3.0:1789760731.207389:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760731.207406:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5810, 5810] 00000008:00010000:3.0:1789760731.207420:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760731.207425:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760731.207435:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23797760, 23801856) append 0 append lockpos 0 00000080:00200000:3.0:1789760731.207447:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23797760=0x16b2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760731.207492:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760731.207495:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23801856 00000020:00010000:3.0:1789760731.207513:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760731.207516:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.207519:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.207522:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760731.207525:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760731.207532:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760731.207537:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760731.207551:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760731.207626:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aea00 - object 0x240000400:5 offset 23797760<>23801856 00000100:00100000:2.0:1789760731.208468:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512603520:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760731.245385:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760731.245396:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760731.245405:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760731.245422:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760731.245444:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512603520:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760731.245529:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5810, 5810] 00000080:00200000:3.0:1789760731.245543:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760731.245548:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760731.245560:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23801856 i_blocks 47111 00000020:00200000:3.0:1789760731.245563:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760731.245570:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760731.245596:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23801856, count: 4096 00000080:00200000:3.0:1789760731.245601:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23801856, bytes: 4096 00000080:00200000:3.0:1789760731.245609:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760731.245616:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760731.245618:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23801856, 23805952) 0 0 00000080:00200000:3.0:1789760731.245622:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760731.245634:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5811, 5811] 00020000:00200000:3.0:1789760731.245640:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23801856 chunk: [23801856, 23805952) 23805952, 4096 00020000:00200000:3.0:1789760731.245644:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760731.245660:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5811, 5811] 00000008:00010000:3.0:1789760731.245673:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760731.245678:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760731.245686:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23801856, 23805952) append 0 append lockpos 0 00000080:00200000:3.0:1789760731.245697:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23801856=0x16b3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760731.245741:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760731.245744:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23805952 00000020:00010000:3.0:1789760731.245750:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760731.245753:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.245756:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.245759:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760731.245762:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760731.245767:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760731.245772:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760731.245786:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760731.245861:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af100 - object 0x240000400:5 offset 23801856<>23805952 00000100:00100000:2.0:1789760731.246353:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760731.246369:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512603648:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760731.295145:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973130 gen 3 00000008:00010000:2.0:1789760731.295203:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760731.295210:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760731.295218:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760731.295236:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760731.295259:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512603648:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760731.295341:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5811, 5811] 00000080:00200000:3.0:1789760731.295354:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760731.295359:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760731.295372:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23805952 i_blocks 47111 00000020:00200000:3.0:1789760731.295375:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760731.295382:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760731.295407:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23805952, count: 4096 00000080:00200000:3.0:1789760731.295413:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23805952, bytes: 4096 00000080:00200000:3.0:1789760731.295421:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760731.295429:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760731.295431:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23805952, 23810048) 0 0 00000080:00200000:3.0:1789760731.295435:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760731.295447:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5812, 5812] 00020000:00200000:3.0:1789760731.295453:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23805952 chunk: [23805952, 23810048) 23810048, 4096 00020000:00200000:3.0:1789760731.295456:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760731.295472:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5812, 5812] 00000008:00010000:3.0:1789760731.295485:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760731.295489:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760731.295509:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23805952, 23810048) append 0 append lockpos 0 00000080:00200000:3.0:1789760731.295521:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23805952=0x16b4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760731.295561:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760731.295563:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23810048 00000020:00010000:3.0:1789760731.295569:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760731.295572:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.295574:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.295578:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760731.295580:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760731.295586:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760731.295591:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760731.295605:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760731.295681:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad180 - object 0x240000400:5 offset 23805952<>23810048 00000100:00100000:2.0:1789760731.295994:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512603776:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760731.337199:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760731.337210:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760731.337219:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760731.337237:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760731.337260:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512603776:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760731.337338:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5812, 5812] 00000080:00200000:3.0:1789760731.337351:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760731.337356:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760731.337367:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23810048 i_blocks 47111 00000020:00200000:3.0:1789760731.337370:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760731.337377:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760731.337403:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23810048, count: 4096 00000080:00200000:3.0:1789760731.337410:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23810048, bytes: 4096 00000080:00200000:3.0:1789760731.337417:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760731.337424:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760731.337426:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23810048, 23814144) 0 0 00000080:00200000:3.0:1789760731.337430:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760731.337444:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5813, 5813] 00020000:00200000:3.0:1789760731.337451:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23810048 chunk: [23810048, 23814144) 23814144, 4096 00020000:00200000:3.0:1789760731.337455:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760731.337470:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5813, 5813] 00000008:00010000:3.0:1789760731.337485:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760731.337489:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760731.337506:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23810048, 23814144) append 0 append lockpos 0 00000080:00200000:3.0:1789760731.337517:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23810048=0x16b5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760731.337558:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760731.337562:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23814144 00000020:00010000:3.0:1789760731.337568:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760731.337571:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.337574:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.337577:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760731.337580:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760731.337585:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760731.337591:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760731.337604:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760731.337677:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adf80 - object 0x240000400:5 offset 23810048<>23814144 00000100:00100000:2.0:1789760731.337810:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512603904:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760731.373226:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973132 gen 3 00000008:00010000:2.0:1789760731.373273:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760731.373280:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760731.373288:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760731.373302:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760731.373323:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512603904:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760731.373524:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5813, 5813] 00000080:00200000:3.0:1789760731.373537:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760731.373542:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760731.373556:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23814144 i_blocks 47111 00000020:00200000:3.0:1789760731.373559:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760731.373567:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760731.373593:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23814144, count: 4096 00000080:00200000:3.0:1789760731.373599:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23814144, bytes: 4096 00000080:00200000:3.0:1789760731.373606:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760731.373613:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760731.373615:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23814144, 23818240) 0 0 00000080:00200000:3.0:1789760731.373619:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760731.373631:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5814, 5814] 00020000:00200000:3.0:1789760731.373638:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23814144 chunk: [23814144, 23818240) 23818240, 4096 00020000:00200000:3.0:1789760731.373642:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760731.373656:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5814, 5814] 00000008:00010000:3.0:1789760731.373668:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760731.373673:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760731.373682:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23814144, 23818240) append 0 append lockpos 0 00000080:00200000:3.0:1789760731.373693:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23814144=0x16b6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760731.373735:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760731.373738:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23818240 00000020:00010000:3.0:1789760731.373745:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760731.373748:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.373751:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.373754:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760731.373756:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760731.373762:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760731.373768:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760731.373782:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760731.373856:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad880 - object 0x240000400:5 offset 23814144<>23818240 00000100:00100000:2.0:1789760731.374116:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512604032:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760731.457850:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973133 gen 3 00000008:00010000:2.0:1789760731.457888:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760731.457929:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760731.457939:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760731.457957:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:2.0:1789760731.457980:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512604032:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760731.458176:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5814, 5814] 00000080:00200000:3.0:1789760731.458189:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760731.458193:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760731.458205:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23818240 i_blocks 47111 00000020:00200000:3.0:1789760731.458209:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760731.458217:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760731.458242:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23818240, count: 4096 00000080:00200000:3.0:1789760731.458249:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23818240, bytes: 4096 00000080:00200000:3.0:1789760731.458256:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760731.458264:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760731.458266:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23818240, 23822336) 0 0 00000080:00200000:3.0:1789760731.458270:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760731.458285:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5815, 5815] 00020000:00200000:3.0:1789760731.458292:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23818240 chunk: [23818240, 23822336) 23822336, 4096 00020000:00200000:3.0:1789760731.458296:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760731.458312:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5815, 5815] 00000008:00010000:3.0:1789760731.458326:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760731.458331:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760731.458341:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23818240, 23822336) append 0 append lockpos 0 00000080:00200000:3.0:1789760731.458351:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23818240=0x16b7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760731.458397:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760731.458400:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23822336 00000020:00010000:3.0:1789760731.458408:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760731.458411:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.458413:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.458416:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760731.458419:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760731.458425:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760731.458430:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760731.458444:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760731.458529:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 23818240<>23822336 00000100:00100000:3.0:1789760731.458573:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760731.458585:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512604160:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760731.505169:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973134 gen 3 00000008:00010000:2.0:1789760731.505202:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760731.505208:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760731.505216:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760731.505230:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760731.505252:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512604160:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760731.505262:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5815, 5815] 00000080:00200000:3.0:1789760731.505277:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760731.505282:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760731.505295:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23822336 i_blocks 47111 00000020:00200000:3.0:1789760731.505298:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760731.505308:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760731.505332:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23822336, count: 4096 00000080:00200000:3.0:1789760731.505338:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23822336, bytes: 4096 00000080:00200000:3.0:1789760731.505346:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760731.505353:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760731.505355:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23822336, 23826432) 0 0 00000080:00200000:3.0:1789760731.505359:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760731.505372:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5816, 5816] 00020000:00200000:3.0:1789760731.505379:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23822336 chunk: [23822336, 23826432) 23826432, 4096 00020000:00200000:3.0:1789760731.505383:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760731.505396:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5816, 5816] 00000008:00010000:3.0:1789760731.505408:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760731.505412:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760731.505421:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23822336, 23826432) append 0 append lockpos 0 00000080:00200000:3.0:1789760731.505430:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23822336=0x16b8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760731.505480:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760731.505483:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23826432 00000020:00010000:3.0:1789760731.505489:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760731.505492:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.505495:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.505507:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760731.505510:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760731.505516:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760731.505521:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760731.505534:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760731.505614:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae680 - object 0x240000400:5 offset 23822336<>23826432 00000100:00100000:2.0:1789760731.505675:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760731.505691:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512604288:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760731.552897:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973135 gen 3 00000008:00010000:2.0:1789760731.552959:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760731.552965:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760731.552975:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760731.552995:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760731.553018:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512604288:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760731.553394:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5816, 5816] 00000080:00200000:3.0:1789760731.553408:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760731.553413:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760731.553425:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23826432 i_blocks 47111 00000020:00200000:3.0:1789760731.553429:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760731.553437:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760731.553462:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23826432, count: 4096 00000080:00200000:3.0:1789760731.553468:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23826432, bytes: 4096 00000080:00200000:3.0:1789760731.553479:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760731.553486:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760731.553488:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23826432, 23830528) 0 0 00000080:00200000:3.0:1789760731.553492:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760731.553517:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5817, 5817] 00020000:00200000:3.0:1789760731.553524:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23826432 chunk: [23826432, 23830528) 23830528, 4096 00020000:00200000:3.0:1789760731.553528:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760731.553660:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5817, 5817] 00000008:00010000:3.0:1789760731.553679:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760731.553683:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760731.553693:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23826432, 23830528) append 0 append lockpos 0 00000080:00200000:3.0:1789760731.553704:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23826432=0x16b9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760731.553749:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760731.553752:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23830528 00000020:00010000:3.0:1789760731.553758:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760731.553762:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.553765:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.553769:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760731.553772:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760731.553778:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760731.553784:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760731.553797:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760731.553877:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 23826432<>23830528 00000100:00100000:2.0:1789760731.554015:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512604416:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760731.597256:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760731.597267:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760731.597277:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760731.597290:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760731.597316:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512604416:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760731.597615:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5817, 5817] 00000080:00200000:3.0:1789760731.597628:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760731.597632:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760731.597645:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23830528 i_blocks 47111 00000020:00200000:3.0:1789760731.597648:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760731.597655:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760731.597680:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23830528, count: 4096 00000080:00200000:3.0:1789760731.597687:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23830528, bytes: 4096 00000080:00200000:3.0:1789760731.597694:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760731.597701:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760731.597703:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23830528, 23834624) 0 0 00000080:00200000:3.0:1789760731.597707:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760731.597720:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5818, 5818] 00020000:00200000:3.0:1789760731.597727:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23830528 chunk: [23830528, 23834624) 23834624, 4096 00020000:00200000:3.0:1789760731.597730:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760731.597744:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5818, 5818] 00000008:00010000:3.0:1789760731.597763:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760731.597771:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760731.597780:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23830528, 23834624) append 0 append lockpos 0 00000080:00200000:3.0:1789760731.597792:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23830528=0x16ba000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760731.597834:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760731.597837:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23834624 00000020:00010000:3.0:1789760731.597843:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760731.597846:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.597848:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.597851:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760731.597854:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760731.597860:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760731.597865:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760731.597880:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760731.597988:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af800 - object 0x240000400:5 offset 23830528<>23834624 00000100:00100000:2.0:1789760731.598044:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512604544:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760731.656710:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973137 gen 3 00000008:00010000:2.0:1789760731.656763:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760731.656769:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760731.656778:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760731.656802:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760731.656829:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512604544:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760731.656976:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5818, 5818] 00000080:00200000:3.0:1789760731.656990:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760731.656996:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760731.657010:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23834624 i_blocks 47111 00000020:00200000:3.0:1789760731.657160:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760731.657171:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760731.657198:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23834624, count: 4096 00000080:00200000:3.0:1789760731.657205:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23834624, bytes: 4096 00000080:00200000:3.0:1789760731.657216:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760731.657224:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760731.657226:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23834624, 23838720) 0 0 00000080:00200000:3.0:1789760731.657230:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760731.657243:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5819, 5819] 00020000:00200000:3.0:1789760731.657250:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23834624 chunk: [23834624, 23838720) 23838720, 4096 00020000:00200000:3.0:1789760731.657254:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760731.657268:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5819, 5819] 00000008:00010000:3.0:1789760731.657283:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760731.657288:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760731.657298:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23834624, 23838720) append 0 append lockpos 0 00000080:00200000:3.0:1789760731.657309:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23834624=0x16bb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760731.657358:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760731.657361:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23838720 00000020:00010000:3.0:1789760731.657367:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760731.657370:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.657373:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.657376:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760731.657379:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760731.657386:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760731.657392:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760731.657407:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760731.657489:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af800 - object 0x240000400:5 offset 23834624<>23838720 00000100:00100000:2.0:1789760731.657645:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512604672:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760731.777665:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973138 gen 3 00000008:00010000:2.0:1789760731.777704:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760731.777712:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760731.777722:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760731.777742:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:2.0:1789760731.777764:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512604672:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760731.777859:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5819, 5819] 00000080:00200000:3.0:1789760731.777874:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760731.777880:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760731.777895:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23838720 i_blocks 47111 00000020:00200000:3.0:1789760731.777921:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760731.777932:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760731.777960:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23838720, count: 4096 00000080:00200000:3.0:1789760731.777966:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23838720, bytes: 4096 00000080:00200000:3.0:1789760731.777973:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760731.777980:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760731.777982:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23838720, 23842816) 0 0 00000080:00200000:3.0:1789760731.777985:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760731.778016:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5820, 5820] 00020000:00200000:3.0:1789760731.778022:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23838720 chunk: [23838720, 23842816) 23842816, 4096 00020000:00200000:3.0:1789760731.778026:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760731.778042:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5820, 5820] 00000008:00010000:3.0:1789760731.778055:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760731.778060:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760731.778070:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23838720, 23842816) append 0 append lockpos 0 00000080:00200000:3.0:1789760731.778081:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23838720=0x16bc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760731.778126:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760731.778128:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23842816 00000020:00010000:3.0:1789760731.778135:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760731.778138:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.778141:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.778144:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760731.778146:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760731.778152:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760731.778157:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760731.778169:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760731.778246:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 23838720<>23842816 00000100:00100000:2.0:1789760731.778302:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512604800:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760731.843767:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973139 gen 3 00000008:00010000:2.0:1789760731.843814:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760731.843820:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760731.843829:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760731.843850:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760731.843881:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512604800:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760731.844003:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5820, 5820] 00000080:00200000:3.0:1789760731.844016:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760731.844021:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760731.844033:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23842816 i_blocks 47111 00000020:00200000:3.0:1789760731.844037:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760731.844044:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760731.844071:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23842816, count: 4096 00000080:00200000:3.0:1789760731.844077:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23842816, bytes: 4096 00000080:00200000:3.0:1789760731.844086:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760731.844094:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760731.844096:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23842816, 23846912) 0 0 00000080:00200000:3.0:1789760731.844099:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760731.844116:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5821, 5821] 00020000:00200000:3.0:1789760731.844122:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23842816 chunk: [23842816, 23846912) 23846912, 4096 00020000:00200000:3.0:1789760731.844126:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760731.844144:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5821, 5821] 00000008:00010000:3.0:1789760731.844158:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760731.844163:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760731.844172:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23842816, 23846912) append 0 append lockpos 0 00000080:00200000:3.0:1789760731.844182:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23842816=0x16bd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760731.844225:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760731.844228:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23846912 00000020:00010000:3.0:1789760731.844234:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760731.844237:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.844240:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.844243:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760731.844245:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760731.844251:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760731.844257:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760731.844270:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760731.844344:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae680 - object 0x240000400:5 offset 23842816<>23846912 00000100:00100000:2.0:1789760731.844651:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512604928:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760731.919575:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973140 gen 3 00000008:00010000:2.0:1789760731.919613:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760731.919619:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760731.919635:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760731.919649:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760731.919669:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512604928:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760731.919791:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5821, 5821] 00000080:00200000:3.0:1789760731.919803:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760731.919808:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760731.919820:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23846912 i_blocks 47111 00000020:00200000:3.0:1789760731.919823:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760731.919831:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760731.919856:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23846912, count: 4096 00000080:00200000:3.0:1789760731.919862:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23846912, bytes: 4096 00000080:00200000:3.0:1789760731.919869:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760731.919876:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760731.919877:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23846912, 23851008) 0 0 00000080:00200000:3.0:1789760731.919881:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760731.919894:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5822, 5822] 00020000:00200000:3.0:1789760731.919919:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23846912 chunk: [23846912, 23851008) 23851008, 4096 00020000:00200000:3.0:1789760731.919922:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760731.919937:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5822, 5822] 00000008:00010000:3.0:1789760731.919950:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760731.919955:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760731.919963:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23846912, 23851008) append 0 append lockpos 0 00000080:00200000:3.0:1789760731.919973:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23846912=0x16be000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760731.920026:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760731.920029:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23851008 00000020:00010000:3.0:1789760731.920035:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760731.920038:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.920041:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.920044:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760731.920047:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760731.920053:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760731.920059:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760731.920072:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760731.920147:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 23846912<>23851008 00000100:00100000:3.0:1789760731.920186:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760731.920198:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512605056:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760731.935214:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:2.0:1789760731.972577:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760731.972588:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760731.972598:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760731.972617:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760731.972640:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512605056:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760731.972763:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5822, 5822] 00000080:00200000:3.0:1789760731.972776:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760731.972780:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760731.972791:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23851008 i_blocks 47111 00000020:00200000:3.0:1789760731.972795:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760731.972803:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760731.972828:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23851008, count: 4096 00000080:00200000:3.0:1789760731.972834:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23851008, bytes: 4096 00000080:00200000:3.0:1789760731.972843:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760731.972850:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760731.972851:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23851008, 23855104) 0 0 00000080:00200000:3.0:1789760731.972855:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760731.972869:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5823, 5823] 00020000:00200000:3.0:1789760731.972875:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23851008 chunk: [23851008, 23855104) 23855104, 4096 00020000:00200000:3.0:1789760731.972879:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760731.972935:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5823, 5823] 00000008:00010000:3.0:1789760731.972951:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760731.972956:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760731.972967:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23851008, 23855104) append 0 append lockpos 0 00000080:00200000:3.0:1789760731.972989:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23851008=0x16bf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760731.973030:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760731.973032:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23855104 00000020:00010000:3.0:1789760731.973039:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760731.973042:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.973044:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760731.973047:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760731.973050:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760731.973056:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760731.973061:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760731.973075:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760731.973151:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad880 - object 0x240000400:5 offset 23851008<>23855104 00000100:00100000:2.0:1789760731.973361:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760731.973376:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512605184:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760732.013369:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973142 gen 3 00000008:00010000:2.0:1789760732.013416:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760732.013424:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760732.013433:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760732.013453:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760732.013476:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512605184:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760732.013631:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5823, 5823] 00000080:00200000:3.0:1789760732.013645:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760732.013651:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760732.013663:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23855104 i_blocks 47111 00000020:00200000:3.0:1789760732.013667:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760732.013675:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760732.013703:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23855104, count: 4096 00000080:00200000:3.0:1789760732.013709:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23855104, bytes: 4096 00000080:00200000:3.0:1789760732.013716:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760732.013723:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760732.013725:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23855104, 23859200) 0 0 00000080:00200000:3.0:1789760732.013729:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760732.013744:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5824, 5824] 00020000:00200000:3.0:1789760732.013752:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23855104 chunk: [23855104, 23859200) 23859200, 4096 00020000:00200000:3.0:1789760732.013756:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760732.013770:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5824, 5824] 00000008:00010000:3.0:1789760732.013783:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760732.013788:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760732.013797:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23855104, 23859200) append 0 append lockpos 0 00000080:00200000:3.0:1789760732.013808:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23855104=0x16c0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760732.013845:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760732.013847:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23859200 00000020:00010000:3.0:1789760732.013853:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760732.013857:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.013859:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.013863:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760732.013865:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760732.013871:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760732.013876:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760732.014049:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760732.014120:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adf80 - object 0x240000400:5 offset 23855104<>23859200 00000100:00100000:2.0:1789760732.014174:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512605312:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760732.067536:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973143 gen 3 00000008:00010000:2.0:1789760732.067577:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760732.067583:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760732.067593:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760732.067611:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760732.067635:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512605312:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760732.067643:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5824, 5824] 00000080:00200000:3.0:1789760732.067658:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760732.067663:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760732.067676:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23859200 i_blocks 47111 00000020:00200000:3.0:1789760732.067680:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760732.067688:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760732.067716:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23859200, count: 4096 00000080:00200000:3.0:1789760732.067723:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23859200, bytes: 4096 00000080:00200000:3.0:1789760732.067731:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760732.067738:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760732.067740:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23859200, 23863296) 0 0 00000080:00200000:3.0:1789760732.067743:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760732.067756:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5825, 5825] 00020000:00200000:3.0:1789760732.067762:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23859200 chunk: [23859200, 23863296) 23863296, 4096 00020000:00200000:3.0:1789760732.067766:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760732.067780:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5825, 5825] 00000008:00010000:3.0:1789760732.067792:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760732.067797:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760732.067805:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23859200, 23863296) append 0 append lockpos 0 00000080:00200000:3.0:1789760732.067815:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23859200=0x16c1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760732.067859:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760732.067861:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23863296 00000020:00010000:3.0:1789760732.067868:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760732.067870:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.067873:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.067876:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760732.067878:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760732.067884:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760732.068040:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760732.068060:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760732.068132:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad180 - object 0x240000400:5 offset 23859200<>23863296 00000100:00100000:2.0:1789760732.068200:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512605440:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760732.117394:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973144 gen 3 00000008:00010000:2.0:1789760732.117428:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760732.117435:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760732.117444:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760732.117463:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760732.117484:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512605440:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760732.117591:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5825, 5825] 00000080:00200000:3.0:1789760732.117604:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760732.117609:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760732.117622:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23863296 i_blocks 47111 00000020:00200000:3.0:1789760732.117625:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760732.117633:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760732.117659:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23863296, count: 4096 00000080:00200000:3.0:1789760732.117664:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23863296, bytes: 4096 00000080:00200000:3.0:1789760732.117672:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760732.117679:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760732.117681:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23863296, 23867392) 0 0 00000080:00200000:3.0:1789760732.117685:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760732.117699:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5826, 5826] 00020000:00200000:3.0:1789760732.117707:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23863296 chunk: [23863296, 23867392) 23867392, 4096 00020000:00200000:3.0:1789760732.117710:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760732.117724:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5826, 5826] 00000008:00010000:3.0:1789760732.117736:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760732.117741:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760732.117749:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23863296, 23867392) append 0 append lockpos 0 00000080:00200000:3.0:1789760732.117759:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23863296=0x16c2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760732.117801:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760732.117803:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23867392 00000020:00010000:3.0:1789760732.117809:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760732.117811:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.117814:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.117817:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760732.117819:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760732.117824:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760732.117828:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760732.117841:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760732.118076:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af100 - object 0x240000400:5 offset 23863296<>23867392 00000100:00100000:2.0:1789760732.118268:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512605568:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760732.161506:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973145 gen 3 00000008:00010000:2.0:1789760732.161541:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760732.161548:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760732.161556:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760732.161638:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:2.0:1789760732.161661:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512605568:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760732.161804:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5826, 5826] 00000080:00200000:3.0:1789760732.161816:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760732.161821:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760732.161831:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23867392 i_blocks 47111 00000020:00200000:3.0:1789760732.161834:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760732.161841:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760732.161867:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23867392, count: 4096 00000080:00200000:3.0:1789760732.161873:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23867392, bytes: 4096 00000080:00200000:3.0:1789760732.161881:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760732.162029:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760732.162031:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23867392, 23871488) 0 0 00000080:00200000:3.0:1789760732.162035:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760732.162049:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5827, 5827] 00020000:00200000:3.0:1789760732.162055:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23867392 chunk: [23867392, 23871488) 23871488, 4096 00020000:00200000:3.0:1789760732.162059:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760732.162074:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5827, 5827] 00000008:00010000:3.0:1789760732.162087:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760732.162092:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760732.162101:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23867392, 23871488) append 0 append lockpos 0 00000080:00200000:3.0:1789760732.162111:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23867392=0x16c3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760732.162319:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760732.162322:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23871488 00000020:00010000:3.0:1789760732.162329:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760732.162331:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.162334:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.162336:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760732.162339:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760732.162344:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760732.162349:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760732.162362:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760732.162437:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aea00 - object 0x240000400:5 offset 23867392<>23871488 00000100:00100000:2.0:1789760732.162506:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512605696:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760732.218658:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973146 gen 3 00000008:00010000:2.0:1789760732.218695:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760732.218701:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760732.218711:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760732.218729:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760732.218753:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512605696:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760732.218850:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5827, 5827] 00000080:00200000:3.0:1789760732.218863:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760732.218867:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760732.218878:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23871488 i_blocks 47111 00000020:00200000:3.0:1789760732.218881:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760732.218890:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760732.218939:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23871488, count: 4096 00000080:00200000:3.0:1789760732.218945:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23871488, bytes: 4096 00000080:00200000:3.0:1789760732.218953:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760732.218960:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760732.218962:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23871488, 23875584) 0 0 00000080:00200000:3.0:1789760732.218965:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760732.218978:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5828, 5828] 00020000:00200000:3.0:1789760732.218985:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23871488 chunk: [23871488, 23875584) 23875584, 4096 00020000:00200000:3.0:1789760732.218988:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760732.219003:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5828, 5828] 00000008:00010000:3.0:1789760732.219018:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760732.219022:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760732.219044:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23871488, 23875584) append 0 append lockpos 0 00000080:00200000:3.0:1789760732.219054:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23871488=0x16c4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760732.219099:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760732.219102:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23875584 00000020:00010000:3.0:1789760732.219109:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760732.219112:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.219115:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.219118:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760732.219121:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760732.219127:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760732.219143:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760732.219158:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760732.219239:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac380 - object 0x240000400:5 offset 23871488<>23875584 00000100:00100000:2.0:1789760732.219400:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512605824:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760732.253538:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973147 gen 3 00000008:00010000:2.0:1789760732.253695:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760732.253702:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760732.253712:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760732.253772:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760732.253798:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512605824:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760732.253871:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5828, 5828] 00000080:00200000:3.0:1789760732.253883:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760732.253888:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760732.253921:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23875584 i_blocks 47111 00000020:00200000:3.0:1789760732.253925:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760732.253933:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760732.253958:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23875584, count: 4096 00000080:00200000:3.0:1789760732.253965:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23875584, bytes: 4096 00000080:00200000:3.0:1789760732.253972:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760732.253979:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760732.253981:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23875584, 23879680) 0 0 00000080:00200000:3.0:1789760732.253985:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760732.253999:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5829, 5829] 00020000:00200000:3.0:1789760732.254006:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23875584 chunk: [23875584, 23879680) 23879680, 4096 00020000:00200000:3.0:1789760732.254010:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760732.254025:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5829, 5829] 00000008:00010000:3.0:1789760732.254039:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760732.254043:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760732.254053:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23875584, 23879680) append 0 append lockpos 0 00000080:00200000:3.0:1789760732.254063:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23875584=0x16c5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760732.254105:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760732.254107:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23879680 00000020:00010000:3.0:1789760732.254113:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760732.254116:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.254119:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.254122:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760732.254125:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760732.254130:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760732.254135:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760732.254218:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760732.254293:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac000 - object 0x240000400:5 offset 23875584<>23879680 00000100:00100000:2.0:1789760732.254351:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512605952:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760732.288047:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973148 gen 3 00000008:00010000:2.0:1789760732.288085:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760732.288091:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760732.288100:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760732.288118:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760732.288141:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512605952:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760732.288248:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5829, 5829] 00000080:00200000:3.0:1789760732.288261:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760732.288266:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760732.288279:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23879680 i_blocks 47111 00000020:00200000:3.0:1789760732.288282:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760732.288290:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760732.288316:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23879680, count: 4096 00000080:00200000:3.0:1789760732.288322:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23879680, bytes: 4096 00000080:00200000:3.0:1789760732.288329:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760732.288338:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760732.288340:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23879680, 23883776) 0 0 00000080:00200000:3.0:1789760732.288343:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760732.288357:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5830, 5830] 00020000:00200000:3.0:1789760732.288364:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23879680 chunk: [23879680, 23883776) 23883776, 4096 00020000:00200000:3.0:1789760732.288368:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760732.288382:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5830, 5830] 00000008:00010000:3.0:1789760732.288396:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760732.288400:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760732.288409:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23879680, 23883776) append 0 append lockpos 0 00000080:00200000:3.0:1789760732.288419:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23879680=0x16c6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760732.288455:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760732.288458:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23883776 00000020:00010000:3.0:1789760732.288464:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760732.288467:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.288470:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.288473:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760732.288476:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760732.288482:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760732.288487:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760732.288549:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:2.0:1789760732.288626:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 23879680<>23883776 00000100:00100000:2.0:1789760732.288662:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760732.288675:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512606080:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760732.324592:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973149 gen 3 00000008:00010000:2.0:1789760732.324629:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760732.324636:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760732.324645:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760732.324663:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760732.324686:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512606080:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760732.324894:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5830, 5830] 00000080:00200000:3.0:1789760732.324942:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760732.324948:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760732.324960:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23883776 i_blocks 47111 00000020:00200000:3.0:1789760732.324963:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760732.324971:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760732.324997:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23883776, count: 4096 00000080:00200000:3.0:1789760732.325003:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23883776, bytes: 4096 00000080:00200000:3.0:1789760732.325011:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760732.325018:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760732.325020:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23883776, 23887872) 0 0 00000080:00200000:3.0:1789760732.325024:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760732.325037:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5831, 5831] 00020000:00200000:3.0:1789760732.325043:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23883776 chunk: [23883776, 23887872) 23887872, 4096 00020000:00200000:3.0:1789760732.325047:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760732.325061:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5831, 5831] 00000008:00010000:3.0:1789760732.325074:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760732.325079:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760732.325089:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23883776, 23887872) append 0 append lockpos 0 00000080:00200000:3.0:1789760732.325100:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23883776=0x16c7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760732.325141:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760732.325144:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23887872 00000020:00010000:3.0:1789760732.325151:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760732.325154:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.325157:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.325170:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760732.325174:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760732.325181:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760732.325186:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760732.325200:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760732.325273:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae300 - object 0x240000400:5 offset 23883776<>23887872 00000100:00100000:3.0:1789760732.325308:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760732.325321:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512606208:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760732.375530:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973150 gen 3 00000008:00010000:2.0:1789760732.375570:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760732.375578:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760732.375587:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760732.375606:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:2.0:1789760732.375629:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512606208:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760732.375719:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5831, 5831] 00000080:00200000:3.0:1789760732.375732:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760732.375736:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760732.375749:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23887872 i_blocks 47111 00000020:00200000:3.0:1789760732.375752:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760732.375760:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760732.375787:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23887872, count: 4096 00000080:00200000:3.0:1789760732.375793:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23887872, bytes: 4096 00000080:00200000:3.0:1789760732.375801:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760732.375808:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760732.375810:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23887872, 23891968) 0 0 00000080:00200000:3.0:1789760732.375815:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760732.375828:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5832, 5832] 00020000:00200000:3.0:1789760732.375834:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23887872 chunk: [23887872, 23891968) 23891968, 4096 00020000:00200000:3.0:1789760732.375838:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760732.375853:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5832, 5832] 00000008:00010000:3.0:1789760732.375868:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760732.375872:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760732.375882:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23887872, 23891968) append 0 append lockpos 0 00000080:00200000:3.0:1789760732.375891:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23887872=0x16c8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760732.375972:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760732.375975:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23891968 00000020:00010000:3.0:1789760732.375982:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760732.375985:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.375989:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.375992:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760732.375995:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760732.376014:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760732.376030:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760732.376044:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760732.376124:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af480 - object 0x240000400:5 offset 23887872<>23891968 00000100:00100000:2.0:1789760732.376309:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512606336:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760732.427720:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973151 gen 3 00000008:00010000:2.0:1789760732.427756:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760732.427763:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760732.427773:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760732.427795:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760732.427819:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512606336:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760732.427927:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5832, 5832] 00000080:00200000:3.0:1789760732.427940:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760732.427946:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760732.427956:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23891968 i_blocks 47111 00000020:00200000:3.0:1789760732.427959:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760732.427967:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760732.427993:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23891968, count: 4096 00000080:00200000:3.0:1789760732.427999:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23891968, bytes: 4096 00000080:00200000:3.0:1789760732.428008:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760732.428016:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760732.428018:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23891968, 23896064) 0 0 00000080:00200000:3.0:1789760732.428022:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760732.428035:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5833, 5833] 00020000:00200000:3.0:1789760732.428042:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23891968 chunk: [23891968, 23896064) 23896064, 4096 00020000:00200000:3.0:1789760732.428045:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760732.428059:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5833, 5833] 00000008:00010000:3.0:1789760732.428072:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760732.428077:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760732.428085:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23891968, 23896064) append 0 append lockpos 0 00000080:00200000:3.0:1789760732.428097:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23891968=0x16c9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760732.428149:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760732.428152:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23896064 00000020:00010000:3.0:1789760732.428159:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760732.428162:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.428165:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.428168:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760732.428170:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760732.428176:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760732.428182:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760732.428195:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760732.428408:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac700 - object 0x240000400:5 offset 23891968<>23896064 00000100:00100000:2.0:1789760732.428511:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760732.428525:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512606464:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760732.476416:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760732.476429:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760732.476438:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760732.476458:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760732.476484:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512606464:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760732.476501:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5833, 5833] 00000080:00200000:3.0:1789760732.476516:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760732.476522:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760732.476534:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23896064 i_blocks 47111 00000020:00200000:3.0:1789760732.476538:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760732.476546:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760732.476572:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23896064, count: 4096 00000080:00200000:3.0:1789760732.476578:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23896064, bytes: 4096 00000080:00200000:3.0:1789760732.476585:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760732.476593:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760732.476595:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23896064, 23900160) 0 0 00000080:00200000:3.0:1789760732.476599:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760732.476613:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5834, 5834] 00020000:00200000:3.0:1789760732.476620:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23896064 chunk: [23896064, 23900160) 23900160, 4096 00020000:00200000:3.0:1789760732.476624:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760732.476639:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5834, 5834] 00000008:00010000:3.0:1789760732.476654:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760732.476659:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760732.476670:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23896064, 23900160) append 0 append lockpos 0 00000080:00200000:3.0:1789760732.476682:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23896064=0x16ca000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760732.476726:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760732.476729:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23900160 00000020:00010000:3.0:1789760732.476734:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760732.476738:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.476741:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.476745:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760732.476747:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760732.476754:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760732.476759:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760732.476774:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760732.477100:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad500 - object 0x240000400:5 offset 23896064<>23900160 00000100:00100000:2.0:1789760732.477755:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760732.477774:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512606592:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760732.509844:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973153 gen 3 00000008:00010000:2.0:1789760732.510079:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760732.510086:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760732.510095:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760732.510115:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760732.510138:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512606592:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760732.510449:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5834, 5834] 00000080:00200000:3.0:1789760732.510464:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760732.510469:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760732.510483:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23900160 i_blocks 47111 00000020:00200000:3.0:1789760732.510486:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760732.510505:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760732.510532:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23900160, count: 4096 00000080:00200000:3.0:1789760732.510538:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23900160, bytes: 4096 00000080:00200000:3.0:1789760732.510546:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760732.510553:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760732.510555:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23900160, 23904256) 0 0 00000080:00200000:3.0:1789760732.510559:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760732.510572:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5835, 5835] 00020000:00200000:3.0:1789760732.510578:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23900160 chunk: [23900160, 23904256) 23904256, 4096 00020000:00200000:3.0:1789760732.510582:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760732.510597:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5835, 5835] 00000008:00010000:3.0:1789760732.510612:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760732.510617:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760732.510629:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23900160, 23904256) append 0 append lockpos 0 00000080:00200000:3.0:1789760732.510641:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23900160=0x16cb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760732.510689:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760732.510692:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23904256 00000020:00010000:3.0:1789760732.510700:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760732.510703:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.510705:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760732.510709:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760732.510711:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760732.510718:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760732.510724:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760732.510739:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760732.511352:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 23900160<>23904256 00000100:00100000:2.0:1789760732.511414:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512606720:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760732.541591:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760732.541601:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760732.541611:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760732.541634:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760732.541658:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512606720:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760732.542233:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512606848:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760732.579751:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973155 gen 3 00000008:00010000:1.0:1789760732.579799:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760732.579805:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760732.579814:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760732.579836:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760732.579861:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512606848:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760732.580514:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760732.580529:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512606976:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760732.625571:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760732.625583:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760732.625592:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760732.625614:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760732.625637:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512606976:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760732.626161:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512607104:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760732.679370:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973156 gen 3 00000100:00100000:1.0:1789760732.679399:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005149180 x1876699512607104/t4294973157(4294973157) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760748 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760732.679437:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760732.679443:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760732.679452:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760732.680258:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760732.680285:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512607104:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760732.688308:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512607232:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760732.734279:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973158 gen 3 00000008:00010000:1.0:1789760732.734324:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760732.734330:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760732.734337:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760732.734344:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760732.734361:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512607232:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760732.734830:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512607360:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760732.780500:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760732.780511:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760732.780519:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760732.780541:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760732.780568:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512607360:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760732.781311:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512607488:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760732.809933:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973160 gen 3 00000008:00010000:1.0:1789760732.809978:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760732.809985:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760732.809993:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760732.810062:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760732.810085:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512607488:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760732.811009:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760732.811024:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512607616:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760732.850318:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760732.850330:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760732.850342:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760732.850368:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760732.850396:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512607616:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760732.851039:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512607744:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760732.893085:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973162 gen 3 00000008:00010000:1.0:1789760732.893165:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760732.893174:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760732.893185:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760732.893211:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760732.893240:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512607744:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760732.894239:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760732.894259:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512607872:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760732.964993:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760732.965005:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760732.965014:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760732.965039:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760732.965061:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512607872:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760732.966932:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760732.966957:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512608000:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.047329:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973164 gen 3 00000008:00010000:1.0:1789760733.047373:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760733.047380:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.047389:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.047411:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760733.047435:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512608000:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.048432:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760733.048449:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512608128:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.093817:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973165 gen 3 00000008:00010000:1.0:1789760733.093856:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760733.093864:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.093873:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.093928:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760733.093951:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512608128:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.094764:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760733.094780:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512608256:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.126453:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973166 gen 3 00000008:00010000:1.0:1789760733.126501:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760733.126507:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.126516:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.126538:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760733.126562:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512608256:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.127385:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760733.127402:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512608384:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.201426:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973167 gen 3 00000008:00010000:1.0:1789760733.201466:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760733.201474:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.201499:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.201521:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760733.201556:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512608384:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.202241:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512608512:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.303763:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973168 gen 3 00000008:00010000:1.0:1789760733.303815:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760733.303823:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.303834:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.303856:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760733.303880:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512608512:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.305493:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512608640:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.363168:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973169 gen 3 00000008:00010000:1.0:1789760733.363203:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760733.363210:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.363219:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.363241:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760733.363262:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512608640:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.363812:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760733.363825:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512608768:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.415100:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973170 gen 3 00000008:00010000:1.0:1789760733.415138:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760733.415145:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.415163:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.415183:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760733.415208:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512608768:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.415829:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512608896:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.471858:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973171 gen 3 00000008:00010000:1.0:1789760733.471929:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760733.471936:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.471944:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.472015:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760733.472038:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512608896:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.472692:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512609024:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.507063:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973172 gen 3 00000008:00010000:1.0:1789760733.507097:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760733.507103:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.507112:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.507193:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760733.507216:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512609024:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760733.507508:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5853, 5853] 00000080:00200000:2.0:1789760733.507521:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760733.507527:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760733.507537:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 23977984 i_blocks 47111 00000020:00200000:2.0:1789760733.507541:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760733.507549:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760733.507575:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 23977984, count: 4096 00000080:00200000:2.0:1789760733.507582:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 23977984, bytes: 4096 00000080:00200000:2.0:1789760733.507590:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760733.507597:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760733.507600:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [23977984, 23982080) 0 0 00000080:00200000:2.0:1789760733.507604:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760733.507616:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5854, 5854] 00020000:00200000:2.0:1789760733.507622:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 23977984 chunk: [23977984, 23982080) 23982080, 4096 00020000:00200000:2.0:1789760733.507626:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760733.507642:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5854, 5854] 00000008:00010000:2.0:1789760733.507655:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760733.507661:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760733.507671:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [23977984, 23982080) append 0 append lockpos 0 00000080:00200000:2.0:1789760733.507682:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=23977984=0x16de000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760733.507728:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760733.507730:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 23982080 00000020:00010000:2.0:1789760733.507737:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760733.507740:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.507743:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.507746:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760733.507749:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760733.507754:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760733.507760:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760733.507775:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760733.507863:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 23977984<>23982080 00000100:00100000:2.0:1789760733.508053:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760733.508068:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512609152:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760733.541095:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760733.541108:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760733.541117:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760733.541141:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760733.541174:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512609152:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.541780:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512609280:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.572728:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973174 gen 3 00000008:00010000:1.0:1789760733.572773:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760733.572780:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.572802:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.572822:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760733.572845:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512609280:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760733.573266:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccf00 00000008:00100000:1.0:1789760733.573336:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94700 - object 0x240000400:5 offset 23986176<>23990272 00000100:00100000:1.0:1789760733.573383:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512609408:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760733.613409:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760733.613421:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.613431:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.613454:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760733.613477:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512609408:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.614154:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512609536:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.642277:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973176 gen 3 00000008:00010000:1.0:1789760733.642332:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760733.642340:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.642350:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.642372:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760733.642397:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512609536:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.643973:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760733.643992:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512609664:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.696307:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973177 gen 3 00000008:00010000:1.0:1789760733.696341:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760733.696347:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.696355:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.696376:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760733.696412:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512609664:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.696934:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512609792:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760733.727634:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760733.727646:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.727656:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.727803:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760733.727835:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512609792:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.729553:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512609920:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.764839:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973178 gen 3 00000100:00100000:1.0:1789760733.764870:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005148e00 x1876699512609920/t4294973179(4294973179) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760749 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760733.764929:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760733.764935:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.764953:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.764973:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760733.764994:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512609920:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.765499:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760733.765515:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512610048:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.816674:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973180 gen 3 00000008:00010000:1.0:1789760733.816726:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760733.816734:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.816744:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.816816:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760733.816840:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512610048:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.817411:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512610176:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.865961:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973181 gen 3 00000008:00010000:1.0:1789760733.866002:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760733.866008:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.866016:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.866041:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760733.866064:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512610176:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760733.866090:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5862, 5862] 00000080:00200000:2.0:1789760733.866104:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760733.866109:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760733.866120:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24014848 i_blocks 47111 00000020:00200000:2.0:1789760733.866123:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760733.866131:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760733.866158:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24014848, count: 4096 00000080:00200000:2.0:1789760733.866164:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24014848, bytes: 4096 00000080:00200000:2.0:1789760733.866172:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760733.866178:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760733.866181:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24014848, 24018944) 0 0 00000080:00200000:2.0:1789760733.866184:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760733.866196:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5863, 5863] 00020000:00200000:2.0:1789760733.866203:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24014848 chunk: [24014848, 24018944) 24018944, 4096 00020000:00200000:2.0:1789760733.866207:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760733.866222:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5863, 5863] 00000008:00010000:2.0:1789760733.866238:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760733.866243:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760733.866252:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24014848, 24018944) append 0 append lockpos 0 00000080:00200000:2.0:1789760733.866314:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24014848=0x16e7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760733.866453:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760733.866456:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24018944 00000020:00010000:2.0:1789760733.866463:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760733.866466:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.866468:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760733.866471:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760733.866474:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760733.866488:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760733.866494:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760733.866507:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760733.866588:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 24014848<>24018944 00000100:00100000:3.0:1789760733.866658:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512610304:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760733.913013:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973182 gen 3 00000008:00010000:2.0:1789760733.913050:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760733.913056:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760733.913064:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760733.913149:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760733.913174:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512610304:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760733.913309:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5863, 5863] 00000080:00200000:1.0:1789760733.913323:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760733.913328:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760733.913339:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24018944 i_blocks 47111 00000020:00200000:1.0:1789760733.913343:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760733.913351:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760733.913377:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24018944, count: 4096 00000080:00200000:1.0:1789760733.913385:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24018944, bytes: 4096 00000080:00200000:1.0:1789760733.913394:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760733.913402:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760733.913404:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24018944, 24023040) 0 0 00000080:00200000:1.0:1789760733.913409:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760733.913425:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5864, 5864] 00020000:00200000:1.0:1789760733.913432:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24018944 chunk: [24018944, 24023040) 24023040, 4096 00020000:00200000:1.0:1789760733.913436:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760733.913454:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5864, 5864] 00000008:00010000:1.0:1789760733.913470:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760733.913475:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760733.913497:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24018944, 24023040) append 0 append lockpos 0 00000080:00200000:1.0:1789760733.913509:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24018944=0x16e8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760733.913578:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760733.913586:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24023040 00000020:00010000:1.0:1789760733.913593:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760733.913596:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760733.913599:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760733.913602:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760733.913604:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760733.913610:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760733.913615:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760733.913628:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760733.913710:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94700 - object 0x240000400:5 offset 24018944<>24023040 00000100:00100000:1.0:1789760733.913746:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760733.913761:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512610432:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.973287:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973183 gen 3 00000008:00010000:1.0:1789760733.973327:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760733.973334:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760733.973343:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760733.973366:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760733.973388:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512610432:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760733.974193:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512610560:192.168.203.156@tcp:4:dd.0 00010000:00010000:1.0:1789760733.981976:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760733.983804:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512610816:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760733.983804:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e6cf7267-9b19-462a-a263-14115df15dab:2361:1876699512610688:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760733.983955:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2361:1876699512610944:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760733.988077:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e6cf7267-9b19-462a-a263-14115df15dab:2361:1876699512610688:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760733.988099:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2361:1876699512610944:192.168.203.156@tcp:400:kworker.0 00000100:00100000:2.0:1789760733.990289:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512610816:192.168.203.156@tcp:400:kworker.0 00000100:00100000:1.0:1789760734.047990:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973184 gen 3 00000008:00010000:1.0:1789760734.048026:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760734.048033:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760734.048044:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760734.048067:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760734.048091:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512610560:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760734.048932:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512611072:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760734.100726:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973185 gen 3 00000008:00010000:1.0:1789760734.100764:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760734.100771:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760734.100780:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760734.100802:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760734.100825:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512611072:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760734.102972:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760734.102996:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512611200:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760734.154356:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760734.154368:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760734.154378:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760734.154462:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760734.154509:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512611200:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760734.154636:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5867, 5867] 00000080:00200000:2.0:1789760734.154648:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760734.154653:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760734.154665:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24035328 i_blocks 47111 00000020:00200000:2.0:1789760734.154668:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760734.154676:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760734.154700:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24035328, count: 4096 00000080:00200000:2.0:1789760734.154706:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24035328, bytes: 4096 00000080:00200000:2.0:1789760734.154714:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760734.154721:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760734.154722:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24035328, 24039424) 0 0 00000080:00200000:2.0:1789760734.154727:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760734.154741:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5868, 5868] 00020000:00200000:2.0:1789760734.154747:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24035328 chunk: [24035328, 24039424) 24039424, 4096 00020000:00200000:2.0:1789760734.154751:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760734.154765:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5868, 5868] 00000008:00010000:2.0:1789760734.154778:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760734.154783:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760734.154866:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24035328, 24039424) append 0 append lockpos 0 00000080:00200000:2.0:1789760734.154877:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24035328=0x16ec000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760734.154951:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760734.154954:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24039424 00000020:00010000:2.0:1789760734.154961:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760734.154965:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.154968:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.154971:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760734.154974:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760734.154980:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760734.154986:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760734.155000:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760734.155172:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 24035328<>24039424 00000100:00100000:3.0:1789760734.155273:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512611328:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760734.187395:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973187 gen 3 00000008:00010000:2.0:1789760734.187442:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760734.187449:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760734.187457:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760734.187487:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760734.187510:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512611328:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760734.187972:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5868, 5868] 00000080:00200000:1.0:1789760734.187986:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760734.187992:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760734.188002:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24039424 i_blocks 47111 00000020:00200000:1.0:1789760734.188006:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760734.188014:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760734.188040:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24039424, count: 4096 00000080:00200000:1.0:1789760734.188058:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24039424, bytes: 4096 00000080:00200000:1.0:1789760734.188067:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760734.188075:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760734.188077:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24039424, 24043520) 0 0 00000080:00200000:1.0:1789760734.188081:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760734.188095:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5869, 5869] 00020000:00200000:1.0:1789760734.188102:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24039424 chunk: [24039424, 24043520) 24043520, 4096 00020000:00200000:1.0:1789760734.188106:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760734.188122:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5869, 5869] 00000008:00010000:1.0:1789760734.188136:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760734.188140:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760734.188150:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24039424, 24043520) append 0 append lockpos 0 00000080:00200000:1.0:1789760734.188160:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24039424=0x16ed000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760734.188204:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760734.188207:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24043520 00000020:00010000:1.0:1789760734.188214:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760734.188217:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760734.188220:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760734.188223:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760734.188226:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760734.188232:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760734.188238:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760734.188251:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760734.188327:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94700 - object 0x240000400:5 offset 24039424<>24043520 00000100:00100000:1.0:1789760734.188372:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760734.188387:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512611456:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760734.244397:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973188 gen 3 00000008:00010000:1.0:1789760734.244434:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760734.244441:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760734.244449:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760734.244473:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760734.244505:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512611456:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760734.278497:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760734.278508:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760734.278517:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760734.278540:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760734.278562:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512611584:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760734.279150:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5870, 5870] 00000080:00200000:2.0:1789760734.279173:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760734.279178:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760734.279189:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24047616 i_blocks 47111 00000020:00200000:2.0:1789760734.279193:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760734.279201:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760734.279226:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24047616, count: 4096 00000080:00200000:2.0:1789760734.279232:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24047616, bytes: 4096 00000080:00200000:2.0:1789760734.279240:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760734.279247:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760734.279249:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24047616, 24051712) 0 0 00000080:00200000:2.0:1789760734.279253:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760734.279266:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5871, 5871] 00020000:00200000:2.0:1789760734.279272:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24047616 chunk: [24047616, 24051712) 24051712, 4096 00020000:00200000:2.0:1789760734.279275:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760734.279289:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5871, 5871] 00000008:00010000:2.0:1789760734.279304:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760734.279309:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760734.279318:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24047616, 24051712) append 0 append lockpos 0 00000080:00200000:2.0:1789760734.279329:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24047616=0x16ef000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760734.279374:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760734.279377:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24051712 00000020:00010000:2.0:1789760734.279383:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760734.279386:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.279390:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c6c8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760734.279393:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760734.279396:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760734.279403:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760734.279409:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760734.279423:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760734.279533:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 24047616<>24051712 00000100:00100000:2.0:1789760734.279563:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760734.279577:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512611712:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760734.334687:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973190 gen 3 00000008:00010000:2.0:1789760734.334735:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760734.334743:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760734.334750:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760734.334825:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760734.334854:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512611712:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760734.335526:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512611840:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760734.406186:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973191 gen 3 00000008:00010000:1.0:1789760734.406226:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760734.406234:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760734.406244:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760734.406265:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760734.406287:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512611840:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760734.408769:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760734.408795:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512611968:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760734.475272:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973192 gen 3 00000008:00010000:1.0:1789760734.475310:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760734.475316:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760734.475325:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760734.475428:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760734.475454:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512611968:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760734.475741:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24059904 i_blocks 47111 00000020:00200000:1.0:1789760734.475746:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760734.475754:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760734.475780:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24059904, count: 4096 00000080:00200000:1.0:1789760734.475787:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24059904, bytes: 4096 00000080:00200000:1.0:1789760734.475936:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760734.475945:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760734.475947:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24059904, 24064000) 0 0 00000080:00200000:1.0:1789760734.475951:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760734.475967:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5874, 5874] 00020000:00200000:1.0:1789760734.475975:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24059904 chunk: [24059904, 24064000) 24064000, 4096 00020000:00200000:1.0:1789760734.475978:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760734.475994:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5874, 5874] 00000008:00010000:1.0:1789760734.476009:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760734.476014:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760734.476024:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24059904, 24064000) append 0 append lockpos 0 00000080:00200000:1.0:1789760734.476034:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24059904=0x16f2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760734.476076:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760734.476078:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24064000 00000020:00010000:1.0:1789760734.476085:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760734.476088:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760734.476091:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760734.476094:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760734.476096:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760734.476102:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760734.476108:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760734.476177:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760734.476259:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94700 - object 0x240000400:5 offset 24059904<>24064000 00000100:00100000:1.0:1789760734.476298:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760734.476313:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512612096:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760734.533026:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760734.533037:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760734.533046:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760734.533062:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760734.533083:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512612096:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760734.570371:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5875, 5875] 00000080:00200000:1.0:1789760734.570383:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760734.570388:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760734.570398:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24068096 i_blocks 47111 00000020:00200000:1.0:1789760734.570401:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760734.570408:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760734.570435:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24068096, count: 4096 00000080:00200000:1.0:1789760734.570440:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24068096, bytes: 4096 00000080:00200000:1.0:1789760734.570448:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760734.570454:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760734.570456:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24068096, 24072192) 0 0 00000080:00200000:1.0:1789760734.570460:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760734.570472:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5876, 5876] 00020000:00200000:1.0:1789760734.570488:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24068096 chunk: [24068096, 24072192) 24072192, 4096 00020000:00200000:1.0:1789760734.570492:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760734.570506:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5876, 5876] 00000008:00010000:1.0:1789760734.570520:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760734.570524:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760734.570533:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24068096, 24072192) append 0 append lockpos 0 00000080:00200000:1.0:1789760734.570542:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24068096=0x16f4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760734.570688:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760734.570691:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24072192 00000020:00010000:1.0:1789760734.570698:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760734.570702:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760734.570704:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760734.570708:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760734.570710:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760734.570716:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760734.570722:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760734.570734:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760734.570808:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 24068096<>24072192 00000100:00100000:1.0:1789760734.570842:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760734.570854:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512612352:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760734.634353:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973195 gen 3 00000008:00010000:1.0:1789760734.634401:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760734.634407:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760734.634415:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760734.634501:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760734.634525:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512612352:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760734.637894:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512612480:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760734.685176:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760734.685188:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760734.685198:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760734.685221:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760734.685248:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512612480:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760734.688268:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512612608:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760734.736042:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973197 gen 3 00000008:00010000:1.0:1789760734.736090:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760734.736097:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760734.736107:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760734.736130:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760734.736222:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512612608:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760734.737951:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512612736:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760734.773127:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973198 gen 3 00000008:00010000:1.0:1789760734.773164:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760734.773170:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760734.773179:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760734.773249:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760734.773268:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512612736:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760734.821870:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5880, 5880] 00000080:00200000:1.0:1789760734.821882:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760734.821887:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760734.821938:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24088576 i_blocks 47111 00000020:00200000:1.0:1789760734.821943:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760734.821951:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760734.821987:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24088576, count: 4096 00000080:00200000:1.0:1789760734.821993:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24088576, bytes: 4096 00000080:00200000:1.0:1789760734.822003:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760734.822010:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760734.822012:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24088576, 24092672) 0 0 00000080:00200000:1.0:1789760734.822016:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760734.822028:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5881, 5881] 00020000:00200000:1.0:1789760734.822034:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24088576 chunk: [24088576, 24092672) 24092672, 4096 00020000:00200000:1.0:1789760734.822038:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760734.822053:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5881, 5881] 00000008:00010000:1.0:1789760734.822067:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760734.822072:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760734.822081:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24088576, 24092672) append 0 append lockpos 0 00000080:00200000:1.0:1789760734.822092:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24088576=0x16f9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760734.822136:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760734.822139:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24092672 00000020:00010000:1.0:1789760734.822145:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760734.822148:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760734.822151:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760734.822155:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760734.822158:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760734.822164:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760734.822169:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760734.822182:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760734.822261:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 24088576<>24092672 00000080:00200000:1.0:1789760734.873522:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5881, 5881] 00000080:00200000:1.0:1789760734.873535:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760734.873601:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760734.873613:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24092672 i_blocks 47111 00000020:00200000:1.0:1789760734.873617:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760734.873624:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760734.873649:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24092672, count: 4096 00000080:00200000:1.0:1789760734.873655:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24092672, bytes: 4096 00000080:00200000:1.0:1789760734.873665:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760734.873673:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760734.873675:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24092672, 24096768) 0 0 00000080:00200000:1.0:1789760734.873679:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760734.873690:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5882, 5882] 00020000:00200000:1.0:1789760734.873760:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24092672 chunk: [24092672, 24096768) 24096768, 4096 00020000:00200000:1.0:1789760734.873765:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760734.873780:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5882, 5882] 00000008:00010000:1.0:1789760734.873792:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760734.873797:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760734.873807:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24092672, 24096768) append 0 append lockpos 0 00000080:00200000:1.0:1789760734.873819:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24092672=0x16fa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760734.873863:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760734.873866:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24096768 00000020:00010000:1.0:1789760734.873873:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760734.873876:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760734.873879:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760734.873883:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760734.873886:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760734.873892:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760734.873898:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760734.873962:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760734.874038:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 24092672<>24096768 00000080:00200000:1.0:1789760734.920787:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5882, 5882] 00000080:00200000:1.0:1789760734.920800:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760734.920804:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760734.920816:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24096768 i_blocks 47111 00000020:00200000:1.0:1789760734.920819:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760734.920827:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760734.920894:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24096768, count: 4096 00000080:00200000:1.0:1789760734.920928:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24096768, bytes: 4096 00000080:00200000:1.0:1789760734.920937:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760734.920944:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760734.920946:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24096768, 24100864) 0 0 00000080:00200000:1.0:1789760734.920950:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760734.920962:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5883, 5883] 00020000:00200000:1.0:1789760734.920969:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24096768 chunk: [24096768, 24100864) 24100864, 4096 00020000:00200000:1.0:1789760734.920972:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760734.920987:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5883, 5883] 00000008:00010000:1.0:1789760734.921000:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760734.921006:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760734.921014:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24096768, 24100864) append 0 append lockpos 0 00000080:00200000:1.0:1789760734.921025:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24096768=0x16fb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760734.921069:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760734.921072:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24100864 00000020:00010000:1.0:1789760734.921078:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760734.921082:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760734.921084:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760734.921088:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760734.921091:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760734.921097:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760734.921103:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760734.921348:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760734.921515:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94380 - object 0x240000400:5 offset 24096768<>24100864 00000080:00200000:1.0:1789760734.977735:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5883, 5883] 00000080:00200000:1.0:1789760734.977754:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760734.977761:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760734.977776:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24100864 i_blocks 47111 00000020:00200000:1.0:1789760734.977779:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760734.977789:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760734.977816:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24100864, count: 4096 00000080:00200000:1.0:1789760734.977822:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24100864, bytes: 4096 00000080:00200000:1.0:1789760734.977831:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760734.977838:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760734.977840:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24100864, 24104960) 0 0 00000080:00200000:1.0:1789760734.977844:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760734.977857:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5884, 5884] 00020000:00200000:1.0:1789760734.977864:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24100864 chunk: [24100864, 24104960) 24104960, 4096 00020000:00200000:1.0:1789760734.977868:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760734.977886:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5884, 5884] 00000008:00010000:1.0:1789760734.977934:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760734.977940:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760734.977950:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24100864, 24104960) append 0 append lockpos 0 00000080:00200000:1.0:1789760734.977961:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24100864=0x16fc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760734.978007:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760734.978010:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24104960 00000020:00010000:1.0:1789760734.978016:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760734.978019:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760734.978022:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760734.978025:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760734.978028:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760734.978034:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760734.978040:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760734.978056:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760734.978151:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 24100864<>24104960 00010000:00010000:2.0:1789760735.005343:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1789760735.039046:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5884, 5884] 00000080:00200000:3.0:1789760735.039057:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760735.039062:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760735.039072:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24104960 i_blocks 47111 00000020:00200000:3.0:1789760735.039075:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760735.039082:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760735.039110:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24104960, count: 4096 00000080:00200000:3.0:1789760735.039115:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24104960, bytes: 4096 00000080:00200000:3.0:1789760735.039124:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760735.039131:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760735.039133:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24104960, 24109056) 0 0 00000080:00200000:3.0:1789760735.039136:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760735.039148:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5885, 5885] 00020000:00200000:3.0:1789760735.039154:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24104960 chunk: [24104960, 24109056) 24109056, 4096 00020000:00200000:3.0:1789760735.039158:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760735.039172:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5885, 5885] 00000008:00010000:3.0:1789760735.039186:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760735.039190:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760735.039199:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24104960, 24109056) append 0 append lockpos 0 00000080:00200000:3.0:1789760735.039211:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24104960=0x16fd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760735.039258:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760735.039260:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24109056 00000020:00010000:3.0:1789760735.039267:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760735.039270:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760735.039273:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760735.039276:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760735.039279:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760735.039285:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760735.039291:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760735.039304:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760735.039382:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107afb80 - object 0x240000400:5 offset 24104960<>24109056 00000100:00100000:3.0:1789760735.039423:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760735.039437:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512613504:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760735.089059:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973203 gen 3 00000100:00100000:2.0:1789760735.089088:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc0107afb80 x1876699512613504/t4294973204(4294973204) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760751 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760735.089128:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760735.089133:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760735.089151:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760735.089171:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760735.089207:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512613504:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.089754:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512613632:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.130782:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973205 gen 3 00000008:00010000:1.0:1789760735.130833:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760735.130839:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760735.130849:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760735.131006:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760735.131033:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512613632:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.131826:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512613760:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.184415:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973206 gen 3 00000008:00010000:1.0:1789760735.184451:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760735.184457:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760735.184466:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760735.184494:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760735.184524:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512613760:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.186616:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512613888:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.229938:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973207 gen 3 00000008:00010000:1.0:1789760735.229976:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760735.229981:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760735.229990:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760735.230060:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760735.230083:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512613888:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.230677:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512614016:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.272709:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973208 gen 3 00000008:00010000:1.0:1789760735.272749:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760735.272754:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760735.272763:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760735.272782:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760735.272815:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512614016:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.273468:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512614144:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.319583:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973209 gen 3 00000008:00010000:1.0:1789760735.319619:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760735.319626:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760735.319635:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760735.319658:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760735.319679:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512614144:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760735.352145:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5891, 5891] 00000080:00200000:2.0:1789760735.352160:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760735.352165:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760735.352177:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24133632 i_blocks 49159 00000020:00200000:2.0:1789760735.352181:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760735.352191:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760735.352217:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24133632, count: 4096 00000080:00200000:2.0:1789760735.352224:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24133632, bytes: 4096 00000080:00200000:2.0:1789760735.352232:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760735.352238:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760735.352240:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24133632, 24137728) 0 0 00000080:00200000:2.0:1789760735.352243:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760735.352256:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5892, 5892] 00020000:00200000:2.0:1789760735.352263:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24133632 chunk: [24133632, 24137728) 24137728, 4096 00020000:00200000:2.0:1789760735.352266:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760735.352282:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5892, 5892] 00000008:00010000:2.0:1789760735.352296:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760735.352301:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760735.352310:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24133632, 24137728) append 0 append lockpos 0 00000080:00200000:2.0:1789760735.352320:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24133632=0x1704000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760735.352366:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760735.352368:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24137728 00000020:00010000:2.0:1789760735.352375:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760735.352379:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.352381:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760735.352384:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760735.352387:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760735.352394:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760735.352400:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760735.352414:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760735.352505:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 24133632<>24137728 00000100:00100000:3.0:1789760735.352649:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512614400:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760735.397311:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973211 gen 3 00000008:00010000:2.0:1789760735.397352:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760735.397358:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760735.397366:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760735.397390:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760735.397413:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512614400:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.399222:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512614528:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.454413:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973212 gen 3 00000008:00010000:1.0:1789760735.454449:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760735.454456:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760735.454466:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760735.454493:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760735.454516:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512614528:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.455458:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512614656:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760735.497987:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760735.497997:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760735.498005:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760735.498026:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760735.498048:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512614656:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.499026:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760735.499042:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512614784:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.536140:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973214 gen 3 00000008:00010000:1.0:1789760735.536188:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760735.536195:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760735.536213:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760735.536235:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760735.536258:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512614784:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.537135:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760735.537150:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512614912:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.592147:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973215 gen 3 00000008:00010000:1.0:1789760735.592188:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760735.592194:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760735.592277:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760735.592297:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760735.592320:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512614912:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.593111:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512615040:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.647065:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973216 gen 3 00000008:00010000:1.0:1789760735.647105:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760735.647111:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760735.647186:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760735.647314:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760735.647340:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512615040:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.649703:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512615168:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760735.690328:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760735.690339:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760735.690349:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760735.690424:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760735.690449:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512615168:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.692767:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760735.692792:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512615296:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.744254:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973218 gen 3 00000008:00010000:1.0:1789760735.744305:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760735.744312:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760735.744320:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760735.744341:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760735.744362:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512615296:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.745365:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760735.745383:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512615424:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.782258:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973219 gen 3 00000008:00010000:1.0:1789760735.782294:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760735.782299:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760735.782309:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760735.782329:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760735.782352:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512615424:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.782897:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512615552:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760735.839946:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760735.839958:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760735.839967:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760735.840039:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760735.840064:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512615552:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.840541:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512615680:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.904626:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973221 gen 3 00000008:00010000:1.0:1789760735.904672:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760735.904678:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760735.904686:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760735.904761:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760735.904784:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512615680:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.906701:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512615808:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760735.969629:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760735.969640:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760735.969649:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760735.969670:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760735.969694:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512615808:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760735.970342:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512615936:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.013791:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973223 gen 3 00000008:00010000:1.0:1789760736.013841:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760736.013847:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.013855:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.013873:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760736.013895:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512615936:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.016079:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512616064:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.047961:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973224 gen 3 00000008:00010000:1.0:1789760736.048001:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760736.048007:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.048026:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.048045:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760736.048066:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512616064:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.049414:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512616192:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760736.086152:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760736.086163:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.086172:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.086192:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760736.086216:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512616192:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.086841:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512616320:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.132707:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973226 gen 3 00000008:00010000:1.0:1789760736.132755:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760736.132761:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.132770:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.132799:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760736.132822:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512616320:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.133441:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512616448:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.163115:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973227 gen 3 00000008:00010000:1.0:1789760736.163162:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760736.163169:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.163178:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.163187:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760736.163210:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512616448:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.163796:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512616576:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.193677:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973228 gen 3 00000008:00010000:1.0:1789760736.193713:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760736.193719:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.193728:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.193749:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760736.193771:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512616576:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.194758:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512616704:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.234777:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973229 gen 3 00000008:00010000:1.0:1789760736.234948:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760736.234954:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.234963:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.234982:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760736.235004:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512616704:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.236729:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512616832:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.277790:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973230 gen 3 00000008:00010000:1.0:1789760736.277823:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760736.277829:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.277836:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.277853:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760736.277879:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512616832:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760736.278010:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5911, 5911] 00000080:00200000:2.0:1789760736.278028:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760736.278032:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760736.278042:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24215552 i_blocks 49159 00000020:00200000:2.0:1789760736.278045:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760736.278052:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760736.278075:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24215552, count: 4096 00000080:00200000:2.0:1789760736.278079:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24215552, bytes: 4096 00000080:00200000:2.0:1789760736.278086:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760736.278091:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760736.278093:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24215552, 24219648) 0 0 00000080:00200000:2.0:1789760736.278096:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760736.278107:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5912, 5912] 00020000:00200000:2.0:1789760736.278114:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24215552 chunk: [24215552, 24219648) 24219648, 4096 00020000:00200000:2.0:1789760736.278118:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760736.278133:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5912, 5912] 00000008:00010000:2.0:1789760736.278147:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760736.278152:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760736.278161:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24215552, 24219648) append 0 append lockpos 0 00000080:00200000:2.0:1789760736.278172:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24215552=0x1718000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760736.278219:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760736.278222:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24219648 00000020:00010000:2.0:1789760736.278229:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760736.278231:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.278234:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.278237:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760736.278239:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760736.278245:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760736.278251:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760736.278264:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760736.278338:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 24215552<>24219648 00000100:00100000:2.0:1789760736.278376:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760736.278391:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512616960:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760736.327805:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973231 gen 3 00000008:00010000:2.0:1789760736.327840:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760736.327847:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760736.327855:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760736.327886:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760736.327951:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512616960:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760736.328122:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5912, 5912] 00000080:00200000:1.0:1789760736.328132:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760736.328143:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760736.328154:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24219648 i_blocks 49159 00000020:00200000:1.0:1789760736.328156:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760736.328163:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760736.328186:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24219648, count: 4096 00000080:00200000:1.0:1789760736.328192:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24219648, bytes: 4096 00000080:00200000:1.0:1789760736.328200:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760736.328207:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760736.328209:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24219648, 24223744) 0 0 00000080:00200000:1.0:1789760736.328212:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760736.328224:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5913, 5913] 00020000:00200000:1.0:1789760736.328230:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24219648 chunk: [24219648, 24223744) 24223744, 4096 00020000:00200000:1.0:1789760736.328234:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760736.328253:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5913, 5913] 00000008:00010000:1.0:1789760736.328267:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760736.328271:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760736.328280:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24219648, 24223744) append 0 append lockpos 0 00000080:00200000:1.0:1789760736.328289:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24219648=0x1719000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760736.328335:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760736.328338:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24223744 00000020:00010000:1.0:1789760736.328344:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760736.328347:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760736.328349:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760736.328352:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760736.328355:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760736.328362:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760736.328367:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760736.328380:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760736.328457:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 24219648<>24223744 00000100:00100000:1.0:1789760736.328559:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760736.328573:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512617088:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.374344:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973232 gen 3 00000008:00010000:1.0:1789760736.374374:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760736.374379:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.374385:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.374393:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760736.374410:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512617088:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.379823:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512617216:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.419282:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973233 gen 3 00000008:00010000:1.0:1789760736.419318:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760736.419325:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.419334:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.419358:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760736.419381:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512617216:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760736.419865:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5914, 5914] 00000080:00200000:2.0:1789760736.419876:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760736.419880:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760736.419889:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24227840 i_blocks 49159 00000020:00200000:2.0:1789760736.419891:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760736.419920:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760736.419943:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24227840, count: 4096 00000080:00200000:2.0:1789760736.419950:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24227840, bytes: 4096 00000080:00200000:2.0:1789760736.419957:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760736.419964:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760736.419966:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24227840, 24231936) 0 0 00000080:00200000:2.0:1789760736.419969:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760736.419982:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5915, 5915] 00020000:00200000:2.0:1789760736.419988:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24227840 chunk: [24227840, 24231936) 24231936, 4096 00020000:00200000:2.0:1789760736.419992:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760736.420007:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5915, 5915] 00000008:00010000:2.0:1789760736.420165:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760736.420174:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760736.420185:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24227840, 24231936) append 0 append lockpos 0 00000080:00200000:2.0:1789760736.420196:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24227840=0x171b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760736.420240:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760736.420242:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24231936 00000020:00010000:2.0:1789760736.420249:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760736.420252:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.420254:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.420258:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760736.420261:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760736.420267:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760736.420272:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760736.420287:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760736.420361:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 24227840<>24231936 00000100:00100000:3.0:1789760736.420443:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512617344:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760736.455354:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973234 gen 3 00000008:00010000:2.0:1789760736.455392:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760736.455398:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760736.455408:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760736.455431:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760736.455453:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512617344:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760736.455627:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5915, 5915] 00000080:00200000:1.0:1789760736.455640:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760736.455646:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760736.455657:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24231936 i_blocks 49159 00000020:00200000:1.0:1789760736.455661:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760736.455670:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760736.455694:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24231936, count: 4096 00000080:00200000:1.0:1789760736.455700:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24231936, bytes: 4096 00000080:00200000:1.0:1789760736.455707:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760736.455714:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760736.455716:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24231936, 24236032) 0 0 00000080:00200000:1.0:1789760736.455720:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760736.455733:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5916, 5916] 00020000:00200000:1.0:1789760736.455740:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24231936 chunk: [24231936, 24236032) 24236032, 4096 00020000:00200000:1.0:1789760736.455744:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760736.455758:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5916, 5916] 00000008:00010000:1.0:1789760736.455773:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760736.455778:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760736.455787:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24231936, 24236032) append 0 append lockpos 0 00000080:00200000:1.0:1789760736.455799:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24231936=0x171c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760736.455842:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760736.455845:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24236032 00000020:00010000:1.0:1789760736.455851:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760736.455854:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760736.455856:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760736.455859:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760736.455862:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760736.455868:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760736.455970:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760736.455984:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760736.456055:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 24231936<>24236032 00000100:00100000:1.0:1789760736.456082:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760736.456094:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512617472:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.500126:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973235 gen 3 00000008:00010000:1.0:1789760736.500161:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760736.500168:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.500177:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.500200:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760736.500233:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512617472:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.503406:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512617600:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.541879:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973236 gen 3 00000008:00010000:1.0:1789760736.541949:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760736.541957:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.541966:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.541992:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760736.542019:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512617600:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.542997:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512617728:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.588036:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973237 gen 3 00000008:00010000:1.0:1789760736.588074:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760736.588081:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.588090:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.588112:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760736.588139:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512617728:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.588704:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760736.588719:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512617856:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760736.654757:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760736.654768:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.654777:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.654855:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760736.654880:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512617856:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760736.722581:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5920, 5920] 00000080:00200000:2.0:1789760736.722593:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760736.722598:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760736.722610:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24252416 i_blocks 49159 00000020:00200000:2.0:1789760736.722613:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760736.722621:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760736.722647:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24252416, count: 4096 00000080:00200000:2.0:1789760736.722654:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24252416, bytes: 4096 00000080:00200000:2.0:1789760736.722662:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760736.722669:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760736.722671:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24252416, 24256512) 0 0 00000080:00200000:2.0:1789760736.722675:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760736.722687:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5921, 5921] 00020000:00200000:2.0:1789760736.722693:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24252416 chunk: [24252416, 24256512) 24256512, 4096 00020000:00200000:2.0:1789760736.722697:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760736.722711:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5921, 5921] 00000008:00010000:2.0:1789760736.722724:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760736.722728:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760736.722737:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24252416, 24256512) append 0 append lockpos 0 00000080:00200000:2.0:1789760736.722747:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24252416=0x1721000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760736.722791:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760736.722794:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24256512 00000020:00010000:2.0:1789760736.722801:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760736.722805:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.722808:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760736.722811:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760736.722814:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760736.722821:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760736.722827:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760736.722840:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760736.722949:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 24252416<>24256512 00000100:00100000:2.0:1789760736.722989:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760736.723001:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512618112:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760736.754130:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760736.754140:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760736.754149:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760736.754162:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760736.754186:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512618112:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760736.757713:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5921, 5921] 00000080:00200000:3.0:1789760736.757732:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760736.757737:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760736.757751:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24256512 i_blocks 49159 00000020:00200000:3.0:1789760736.757755:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760736.757766:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760736.757791:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24256512, count: 4096 00000080:00200000:3.0:1789760736.757798:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24256512, bytes: 4096 00000080:00200000:3.0:1789760736.757809:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760736.757815:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760736.757817:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24256512, 24260608) 0 0 00000080:00200000:3.0:1789760736.757821:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760736.757835:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5922, 5922] 00020000:00200000:3.0:1789760736.757841:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24256512 chunk: [24256512, 24260608) 24260608, 4096 00020000:00200000:3.0:1789760736.757845:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760736.757872:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5922, 5922] 00000008:00010000:3.0:1789760736.757888:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760736.757893:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760736.757947:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24256512, 24260608) append 0 append lockpos 0 00000080:00200000:3.0:1789760736.757959:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24256512=0x1722000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760736.758009:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760736.758012:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24260608 00000020:00010000:3.0:1789760736.758018:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760736.758022:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760736.758025:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760736.758029:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760736.758032:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760736.758039:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760736.758045:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760736.758065:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760736.758149:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 24256512<>24260608 00000100:00100000:3.0:1789760736.758187:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760736.758206:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512618240:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760736.800821:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973241 gen 3 00000008:00010000:2.0:1789760736.800878:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760736.800885:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760736.800894:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760736.800940:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760736.800961:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512618240:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.801560:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512618368:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.834356:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973242 gen 3 00000008:00010000:1.0:1789760736.834396:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760736.834402:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.834412:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.834432:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760736.834471:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512618368:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.835067:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512618496:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.871626:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973243 gen 3 00000008:00010000:1.0:1789760736.871668:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760736.871676:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.871685:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.871705:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760736.871729:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512618496:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.872597:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512618624:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.905014:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973244 gen 3 00000008:00010000:1.0:1789760736.905051:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760736.905056:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.905065:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.905087:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760736.905111:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512618624:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.905799:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512618752:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.949251:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973245 gen 3 00000008:00010000:1.0:1789760736.949290:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760736.949296:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.949305:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.949323:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760736.949345:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512618752:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.949969:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512618880:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.978238:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973246 gen 3 00000008:00010000:1.0:1789760736.978273:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760736.978280:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760736.978290:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760736.978312:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760736.978336:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512618880:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760736.978995:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512619008:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.017984:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973247 gen 3 00000008:00010000:1.0:1789760737.018022:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760737.018028:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.018037:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.018058:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760737.018082:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512619008:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.018661:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512619136:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.049430:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973248 gen 3 00000008:00010000:1.0:1789760737.049481:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760737.049488:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.049497:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.049514:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760737.049538:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512619136:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.050504:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512619264:192.168.203.156@tcp:4:dd.0 00010000:00010000:1.0:1789760737.053779:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:1.0:1789760737.086028:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760737.086039:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.086048:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.086069:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760737.086093:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512619264:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.086680:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512619392:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.125616:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973250 gen 3 00000008:00010000:1.0:1789760737.125663:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760737.125669:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.125678:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.125699:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760737.125722:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512619392:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.127822:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512619520:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.166286:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973251 gen 3 00000008:00010000:1.0:1789760737.166327:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760737.166335:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.166347:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.166376:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760737.166402:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512619520:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760737.166513:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5932, 5932] 00000080:00200000:2.0:1789760737.166595:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.166601:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.166613:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24301568 i_blocks 49159 00000020:00200000:2.0:1789760737.166616:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.166624:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.166650:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24301568, count: 4096 00000080:00200000:2.0:1789760737.166657:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24301568, bytes: 4096 00000080:00200000:2.0:1789760737.166667:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.166674:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.166676:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24301568, 24305664) 0 0 00000080:00200000:2.0:1789760737.166681:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.166694:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5933, 5933] 00020000:00200000:2.0:1789760737.166700:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24301568 chunk: [24301568, 24305664) 24305664, 4096 00020000:00200000:2.0:1789760737.166704:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.166720:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5933, 5933] 00000008:00010000:2.0:1789760737.166734:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.166740:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760737.166749:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24301568, 24305664) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.166759:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24301568=0x172d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.166804:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.166806:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24305664 00000020:00010000:2.0:1789760737.166813:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.166816:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.166818:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.166821:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.166823:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.166829:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.166834:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.166858:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760737.166973:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 24301568<>24305664 00000100:00100000:3.0:1789760737.167042:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512619648:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760737.223215:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760737.223226:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760737.223234:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760737.223312:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760737.223337:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512619648:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.224303:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512619776:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.253420:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973252 gen 3 00000100:00100000:1.0:1789760737.253447:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00514b800 x1876699512619776/t4294973253(4294973253) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760753 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760737.253502:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760737.253508:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.253516:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.253536:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760737.253554:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512619776:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.254484:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512619904:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.309399:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973254 gen 3 00000008:00010000:1.0:1789760737.309556:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760737.309563:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.309572:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.309651:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760737.309678:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512619904:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.312550:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512620032:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.366784:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973255 gen 3 00000008:00010000:1.0:1789760737.366820:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760737.366826:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.366835:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.366855:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760737.366877:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512620032:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.368111:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512620160:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.403480:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973256 gen 3 00000008:00010000:1.0:1789760737.403513:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760737.403519:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.403527:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.403542:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760737.403565:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512620160:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.404469:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760737.404484:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512620288:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.463590:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973257 gen 3 00000008:00010000:1.0:1789760737.463629:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760737.463636:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.463646:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.463670:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760737.463694:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512620288:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.464337:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512620416:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.553306:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973258 gen 3 00000008:00010000:1.0:1789760737.553343:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760737.553349:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.553357:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.553381:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760737.553404:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512620416:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.554031:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512620544:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.634356:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973259 gen 3 00000008:00010000:1.0:1789760737.634394:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760737.634400:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.634410:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.634438:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760737.634472:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512620544:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760737.684332:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5941, 5941] 00000080:00200000:2.0:1789760737.684345:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760737.684351:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760737.684363:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24338432 i_blocks 49159 00000020:00200000:2.0:1789760737.684367:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760737.684375:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760737.684402:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24338432, count: 4096 00000080:00200000:2.0:1789760737.684409:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24338432, bytes: 4096 00000080:00200000:2.0:1789760737.684416:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760737.684424:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760737.684427:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24338432, 24342528) 0 0 00000080:00200000:2.0:1789760737.684431:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760737.684444:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5942, 5942] 00020000:00200000:2.0:1789760737.684450:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24338432 chunk: [24338432, 24342528) 24342528, 4096 00020000:00200000:2.0:1789760737.684467:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760737.684482:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5942, 5942] 00000008:00010000:2.0:1789760737.684495:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760737.684500:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760737.684510:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24338432, 24342528) append 0 append lockpos 0 00000080:00200000:2.0:1789760737.684520:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24338432=0x1736000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760737.687521:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760737.687530:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24342528 00000020:00010000:2.0:1789760737.687541:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760737.687545:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.687549:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760737.687553:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760737.687750:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760737.687764:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760737.687773:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760737.687802:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760737.687917:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 24338432<>24342528 00000100:00100000:3.0:1789760737.687997:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512620800:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760737.736762:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973261 gen 3 00000008:00010000:2.0:1789760737.736807:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760737.736813:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760737.736821:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760737.736841:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760737.736868:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512620800:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.737506:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512620928:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.789131:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973262 gen 3 00000008:00010000:1.0:1789760737.789170:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760737.789176:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.789186:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.789205:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760737.789236:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512620928:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.790087:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512621056:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.836385:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973263 gen 3 00000008:00010000:1.0:1789760737.836424:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760737.836431:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.836440:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.836470:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760737.836495:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512621056:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.837609:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512621184:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.881577:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973264 gen 3 00000008:00010000:1.0:1789760737.881614:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760737.881620:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.881629:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.881649:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760737.881671:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512621184:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.882285:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512621312:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760737.913053:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760737.913065:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.913074:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.913096:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760737.913118:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512621312:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.913776:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512621440:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.942372:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973266 gen 3 00000008:00010000:1.0:1789760737.942419:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760737.942426:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.942435:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.942465:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760737.942490:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512621440:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.943431:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512621568:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760737.976008:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760737.976020:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760737.976029:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760737.976047:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760737.976073:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512621568:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760737.976528:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512621696:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760738.011701:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973267 gen 3 00000100:00100000:1.0:1789760738.011732:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005149180 x1876699512621696/t4294973268(4294973268) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760753 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760738.011767:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760738.011773:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.011782:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.011804:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760738.011824:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512621696:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760738.013644:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512621824:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760738.056237:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973269 gen 3 00000008:00010000:1.0:1789760738.056290:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760738.056300:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.056309:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.056351:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000080:00200000:2.0:1789760738.056370:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5950, 5950] 00000100:00100000:1.0:1789760738.056376:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512621824:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760738.056383:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760738.056388:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760738.056399:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24375296 i_blocks 49159 00000020:00200000:2.0:1789760738.056403:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760738.056410:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760738.056434:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24375296, count: 4096 00000080:00200000:2.0:1789760738.056441:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24375296, bytes: 4096 00000080:00200000:2.0:1789760738.056461:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760738.056467:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760738.056469:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24375296, 24379392) 0 0 00000080:00200000:2.0:1789760738.056473:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760738.056485:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5951, 5951] 00020000:00200000:2.0:1789760738.056492:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24375296 chunk: [24375296, 24379392) 24379392, 4096 00020000:00200000:2.0:1789760738.056496:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760738.056510:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5951, 5951] 00000008:00010000:2.0:1789760738.056523:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760738.056528:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760738.056536:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24375296, 24379392) append 0 append lockpos 0 00000080:00200000:2.0:1789760738.056546:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24375296=0x173f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760738.056652:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760738.056655:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24379392 00000020:00010000:2.0:1789760738.056662:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760738.056665:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.056667:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760738.056671:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760738.056674:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760738.056680:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760738.056685:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760738.056698:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760738.056764:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 24375296<>24379392 00000100:00100000:3.0:1789760738.056820:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512621952:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760738.077104:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760738.105549:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973270 gen 3 00000008:00010000:2.0:1789760738.105582:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760738.105589:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760738.105596:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760738.105676:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760738.105701:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512621952:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760738.107935:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512622080:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760738.143392:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973271 gen 3 00000008:00010000:1.0:1789760738.143429:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760738.143435:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.143444:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.143477:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760738.143499:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512622080:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760738.144438:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760738.144462:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512622208:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760738.182731:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973272 gen 3 00000008:00010000:1.0:1789760738.182769:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760738.182776:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.182785:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.182806:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760738.182829:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512622208:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760738.183475:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512622336:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760738.236751:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973273 gen 3 00000008:00010000:1.0:1789760738.236788:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760738.236795:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.236805:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.236829:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760738.236861:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512622336:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760738.237436:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512622464:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760738.290053:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973274 gen 3 00000008:00010000:1.0:1789760738.290091:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760738.290098:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.290108:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.290133:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760738.290155:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512622464:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760738.292230:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512622592:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760738.327458:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973275 gen 3 00000008:00010000:1.0:1789760738.327497:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760738.327513:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760738.327522:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760738.327544:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760738.327567:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512622592:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760738.365228:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5957, 5957] 00000080:00200000:1.0:1789760738.365240:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760738.365245:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760738.365257:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24403968 i_blocks 49159 00000020:00200000:1.0:1789760738.365261:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760738.365269:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760738.365293:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24403968, count: 4096 00000080:00200000:1.0:1789760738.365299:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24403968, bytes: 4096 00000080:00200000:1.0:1789760738.365307:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760738.365314:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760738.365316:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24403968, 24408064) 0 0 00000080:00200000:1.0:1789760738.365320:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760738.365332:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5958, 5958] 00020000:00200000:1.0:1789760738.365339:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24403968 chunk: [24403968, 24408064) 24408064, 4096 00020000:00200000:1.0:1789760738.365342:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760738.365357:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5958, 5958] 00000008:00010000:1.0:1789760738.365370:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760738.365375:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760738.365384:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24403968, 24408064) append 0 append lockpos 0 00000080:00200000:1.0:1789760738.365394:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24403968=0x1746000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760738.365438:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760738.365441:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24408064 00000020:00010000:1.0:1789760738.365461:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760738.365465:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760738.365468:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760738.365471:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760738.365474:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760738.365480:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760738.365487:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760738.365501:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760738.365586:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 24403968<>24408064 00000080:00200000:1.0:1789760738.404401:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5958, 5958] 00000080:00200000:1.0:1789760738.404414:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760738.404419:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760738.404430:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24408064 i_blocks 49159 00000020:00200000:1.0:1789760738.404434:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760738.404443:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760738.404480:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24408064, count: 4096 00000080:00200000:1.0:1789760738.404487:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24408064, bytes: 4096 00000080:00200000:1.0:1789760738.404496:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760738.404502:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760738.404505:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24408064, 24412160) 0 0 00000080:00200000:1.0:1789760738.404508:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760738.404521:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5959, 5959] 00020000:00200000:1.0:1789760738.404528:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24408064 chunk: [24408064, 24412160) 24412160, 4096 00020000:00200000:1.0:1789760738.404532:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760738.404547:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5959, 5959] 00000008:00010000:1.0:1789760738.404560:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760738.404566:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760738.404575:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24408064, 24412160) append 0 append lockpos 0 00000080:00200000:1.0:1789760738.404586:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24408064=0x1747000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760738.404632:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760738.404635:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24412160 00000020:00010000:1.0:1789760738.404642:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760738.404645:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760738.404648:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760738.404651:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760738.404654:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760738.404660:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760738.404666:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760738.404680:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760738.404756:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96680 - object 0x240000400:5 offset 24408064<>24412160 00000080:00200000:1.0:1789760738.448977:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5959, 5959] 00000080:00200000:1.0:1789760738.448988:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760738.448994:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760738.449006:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24412160 i_blocks 49159 00000020:00200000:1.0:1789760738.449010:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760738.449018:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760738.449042:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24412160, count: 4096 00000080:00200000:1.0:1789760738.449048:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24412160, bytes: 4096 00000080:00200000:1.0:1789760738.449057:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760738.449075:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760738.449078:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24412160, 24416256) 0 0 00000080:00200000:1.0:1789760738.449082:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760738.449096:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5960, 5960] 00020000:00200000:1.0:1789760738.449102:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24412160 chunk: [24412160, 24416256) 24416256, 4096 00020000:00200000:1.0:1789760738.449106:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760738.449217:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5960, 5960] 00000008:00010000:1.0:1789760738.449234:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760738.449238:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760738.449247:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24412160, 24416256) append 0 append lockpos 0 00000080:00200000:1.0:1789760738.449258:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24412160=0x1748000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760738.449307:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760738.449310:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24416256 00000020:00010000:1.0:1789760738.449319:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760738.449322:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760738.449325:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760738.449328:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760738.449331:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760738.449338:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760738.449344:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760738.449361:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760738.449438:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97480 - object 0x240000400:5 offset 24412160<>24416256 00000080:00200000:1.0:1789760738.493535:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5960, 5960] 00000080:00200000:1.0:1789760738.493546:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760738.493560:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760738.493571:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24416256 i_blocks 49159 00000020:00200000:1.0:1789760738.493575:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760738.493582:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760738.493606:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24416256, count: 4096 00000080:00200000:1.0:1789760738.493612:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24416256, bytes: 4096 00000080:00200000:1.0:1789760738.493620:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760738.493627:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760738.493629:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24416256, 24420352) 0 0 00000080:00200000:1.0:1789760738.493633:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760738.493644:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5961, 5961] 00020000:00200000:1.0:1789760738.493651:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24416256 chunk: [24416256, 24420352) 24420352, 4096 00020000:00200000:1.0:1789760738.493655:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760738.493669:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5961, 5961] 00000008:00010000:1.0:1789760738.493681:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760738.493686:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760738.493694:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24416256, 24420352) append 0 append lockpos 0 00000080:00200000:1.0:1789760738.493705:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24416256=0x1749000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760738.493751:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760738.493753:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24420352 00000020:00010000:1.0:1789760738.493760:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760738.493764:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760738.493767:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760738.493770:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760738.493772:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760738.493778:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760738.493783:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760738.493796:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760738.495005:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95880 - object 0x240000400:5 offset 24416256<>24420352 00000080:00200000:3.0:1789760738.557015:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5961, 5961] 00000080:00200000:3.0:1789760738.557027:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760738.557032:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760738.557043:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24420352 i_blocks 49159 00000020:00200000:3.0:1789760738.557047:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760738.557055:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760738.557219:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24420352, count: 4096 00000080:00200000:3.0:1789760738.557227:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24420352, bytes: 4096 00000080:00200000:3.0:1789760738.557235:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760738.557242:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760738.557244:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24420352, 24424448) 0 0 00000080:00200000:3.0:1789760738.557248:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760738.557261:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5962, 5962] 00020000:00200000:3.0:1789760738.557268:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24420352 chunk: [24420352, 24424448) 24424448, 4096 00020000:00200000:3.0:1789760738.557271:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760738.557441:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5962, 5962] 00000008:00010000:3.0:1789760738.557467:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760738.557472:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760738.557481:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24420352, 24424448) append 0 append lockpos 0 00000080:00200000:3.0:1789760738.557491:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24420352=0x174a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760738.557756:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760738.557760:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24424448 00000020:00010000:3.0:1789760738.557767:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760738.557770:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760738.557773:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760738.557776:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760738.557779:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760738.557785:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760738.557796:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760738.557814:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760738.557933:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 24420352<>24424448 00000100:00100000:2.0:1789760738.558078:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512623360:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760738.626293:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973281 gen 3 00000008:00010000:2.0:1789760738.626338:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760738.626345:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760738.626355:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760738.626372:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:2.0:1789760738.626395:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512623360:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760738.626552:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5962, 5962] 00000080:00200000:3.0:1789760738.626564:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760738.626569:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760738.626580:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24424448 i_blocks 49159 00000020:00200000:3.0:1789760738.626584:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760738.626592:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760738.626619:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24424448, count: 4096 00000080:00200000:3.0:1789760738.626625:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24424448, bytes: 4096 00000080:00200000:3.0:1789760738.626633:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760738.626639:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760738.626641:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24424448, 24428544) 0 0 00000080:00200000:3.0:1789760738.626645:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760738.626659:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5963, 5963] 00020000:00200000:3.0:1789760738.626666:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24424448 chunk: [24424448, 24428544) 24428544, 4096 00020000:00200000:3.0:1789760738.626669:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760738.628105:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5963, 5963] 00000008:00010000:3.0:1789760738.628127:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760738.628133:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760738.628144:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24424448, 24428544) append 0 append lockpos 0 00000080:00200000:3.0:1789760738.628156:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24424448=0x174b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760738.628211:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760738.628214:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24428544 00000020:00010000:3.0:1789760738.628221:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760738.628224:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760738.628227:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760738.628230:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760738.628233:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760738.628243:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760738.628251:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760738.628276:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760738.628364:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107afb80 - object 0x240000400:5 offset 24424448<>24428544 00000100:00100000:2.0:1789760738.628617:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760738.628638:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512623488:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760738.713495:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973282 gen 3 00000008:00010000:2.0:1789760738.713546:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760738.713553:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760738.713562:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760738.713583:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760738.713606:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512623488:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760738.713772:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5963, 5963] 00000080:00200000:3.0:1789760738.713785:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760738.713790:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760738.713801:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24428544 i_blocks 49159 00000020:00200000:3.0:1789760738.713805:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760738.713813:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760738.713839:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24428544, count: 4096 00000080:00200000:3.0:1789760738.713845:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24428544, bytes: 4096 00000080:00200000:3.0:1789760738.713852:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760738.713859:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760738.713861:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24428544, 24432640) 0 0 00000080:00200000:3.0:1789760738.713865:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760738.713877:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5964, 5964] 00020000:00200000:3.0:1789760738.713884:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24428544 chunk: [24428544, 24432640) 24432640, 4096 00020000:00200000:3.0:1789760738.713887:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760738.713940:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5964, 5964] 00000008:00010000:3.0:1789760738.713956:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760738.713961:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760738.713970:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24428544, 24432640) append 0 append lockpos 0 00000080:00200000:3.0:1789760738.713992:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24428544=0x174c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760738.714034:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760738.714037:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24432640 00000020:00010000:3.0:1789760738.714043:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760738.714047:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760738.714050:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760738.714053:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760738.714055:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760738.714061:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760738.714066:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760738.714086:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760738.714161:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 24428544<>24432640 00000100:00100000:2.0:1789760738.714878:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512623616:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760738.753875:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973283 gen 3 00000008:00010000:2.0:1789760738.753935:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760738.753941:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760738.753949:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760738.753960:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760738.753978:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512623616:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760738.754200:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5964, 5964] 00000080:00200000:3.0:1789760738.754211:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760738.754217:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760738.754228:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24432640 i_blocks 49159 00000020:00200000:3.0:1789760738.754231:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760738.754239:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760738.754264:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24432640, count: 4096 00000080:00200000:3.0:1789760738.754269:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24432640, bytes: 4096 00000080:00200000:3.0:1789760738.754276:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760738.754283:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760738.754285:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24432640, 24436736) 0 0 00000080:00200000:3.0:1789760738.754289:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760738.754301:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5965, 5965] 00020000:00200000:3.0:1789760738.754306:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24432640 chunk: [24432640, 24436736) 24436736, 4096 00020000:00200000:3.0:1789760738.754310:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760738.754335:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5965, 5965] 00000008:00010000:3.0:1789760738.754348:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760738.754352:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760738.754362:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24432640, 24436736) append 0 append lockpos 0 00000080:00200000:3.0:1789760738.754372:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24432640=0x174d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760738.754415:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760738.754417:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24436736 00000020:00010000:3.0:1789760738.754423:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760738.754426:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760738.754429:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760738.754432:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760738.754435:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760738.754440:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760738.754455:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760738.754468:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760738.754540:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad500 - object 0x240000400:5 offset 24432640<>24436736 00000100:00100000:2.0:1789760738.754770:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512623744:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760738.798354:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973284 gen 3 00000008:00010000:2.0:1789760738.798399:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760738.798405:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760738.798413:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760738.798427:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760738.798456:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512623744:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760738.798651:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5965, 5965] 00000080:00200000:3.0:1789760738.798664:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760738.798670:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760738.798683:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24436736 i_blocks 49159 00000020:00200000:3.0:1789760738.798687:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760738.798695:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760738.798721:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24436736, count: 4096 00000080:00200000:3.0:1789760738.798728:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24436736, bytes: 4096 00000080:00200000:3.0:1789760738.798735:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760738.798742:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760738.798744:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24436736, 24440832) 0 0 00000080:00200000:3.0:1789760738.798748:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760738.798761:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5966, 5966] 00020000:00200000:3.0:1789760738.798768:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24436736 chunk: [24436736, 24440832) 24440832, 4096 00020000:00200000:3.0:1789760738.798772:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760738.798791:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5966, 5966] 00000008:00010000:3.0:1789760738.798805:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760738.798809:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760738.798819:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24436736, 24440832) append 0 append lockpos 0 00000080:00200000:3.0:1789760738.798829:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24436736=0x174e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760738.798976:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760738.798979:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24440832 00000020:00010000:3.0:1789760738.798986:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760738.798989:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760738.798992:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760738.798995:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760738.798998:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760738.799004:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760738.799010:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760738.799024:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760738.799111:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac700 - object 0x240000400:5 offset 24436736<>24440832 00000100:00100000:2.0:1789760738.799168:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512623872:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760738.862812:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760738.862823:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760738.862833:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760738.862861:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760738.862885:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512623872:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760738.922963:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5967, 5967] 00000080:00200000:1.0:1789760738.922975:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760738.922980:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760738.922990:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24444928 i_blocks 49159 00000020:00200000:1.0:1789760738.922994:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760738.923002:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760738.923028:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24444928, count: 4096 00000080:00200000:1.0:1789760738.923034:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24444928, bytes: 4096 00000080:00200000:1.0:1789760738.923042:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760738.923049:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760738.923051:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24444928, 24449024) 0 0 00000080:00200000:1.0:1789760738.923055:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760738.923069:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5968, 5968] 00020000:00200000:1.0:1789760738.923076:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24444928 chunk: [24444928, 24449024) 24449024, 4096 00020000:00200000:1.0:1789760738.923080:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760738.923095:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5968, 5968] 00000008:00010000:1.0:1789760738.923108:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760738.923177:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760738.923189:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24444928, 24449024) append 0 append lockpos 0 00000080:00200000:1.0:1789760738.923201:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24444928=0x1750000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760738.923247:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760738.923249:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24449024 00000020:00010000:1.0:1789760738.923256:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760738.923260:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760738.923263:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760738.923340:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760738.923343:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760738.923351:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760738.923357:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760738.923374:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760738.923459:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96d80 - object 0x240000400:5 offset 24444928<>24449024 00000080:00200000:1.0:1789760738.983494:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5968, 5968] 00000080:00200000:1.0:1789760738.983506:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760738.983511:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760738.983523:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24449024 i_blocks 49159 00000020:00200000:1.0:1789760738.983527:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760738.983536:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760738.983563:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24449024, count: 4096 00000080:00200000:1.0:1789760738.983569:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24449024, bytes: 4096 00000080:00200000:1.0:1789760738.983576:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760738.983583:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760738.983585:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24449024, 24453120) 0 0 00000080:00200000:1.0:1789760738.983590:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760738.983602:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5969, 5969] 00020000:00200000:1.0:1789760738.983609:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24449024 chunk: [24449024, 24453120) 24453120, 4096 00020000:00200000:1.0:1789760738.983612:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760738.983628:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5969, 5969] 00000008:00010000:1.0:1789760738.983642:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760738.983646:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760738.983655:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24449024, 24453120) append 0 append lockpos 0 00000080:00200000:1.0:1789760738.983665:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24449024=0x1751000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760738.983707:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760738.983710:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24453120 00000020:00010000:1.0:1789760738.983717:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760738.983720:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760738.983723:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760738.983726:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760738.983729:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760738.983736:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760738.983741:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760738.983755:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760738.983849:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95880 - object 0x240000400:5 offset 24449024<>24453120 00000080:00200000:1.0:1789760739.021122:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5969, 5969] 00000080:00200000:1.0:1789760739.021135:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760739.021140:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760739.021152:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24453120 i_blocks 49159 00000020:00200000:1.0:1789760739.021156:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760739.021163:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760739.021190:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24453120, count: 4096 00000080:00200000:1.0:1789760739.021195:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24453120, bytes: 4096 00000080:00200000:1.0:1789760739.021203:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760739.021210:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760739.021212:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24453120, 24457216) 0 0 00000080:00200000:1.0:1789760739.021215:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760739.021228:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5970, 5970] 00020000:00200000:1.0:1789760739.021235:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24453120 chunk: [24453120, 24457216) 24457216, 4096 00020000:00200000:1.0:1789760739.021238:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760739.021254:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5970, 5970] 00000008:00010000:1.0:1789760739.021269:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760739.021273:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760739.021282:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24453120, 24457216) append 0 append lockpos 0 00000080:00200000:1.0:1789760739.021295:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24453120=0x1752000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760739.021338:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760739.021341:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24457216 00000020:00010000:1.0:1789760739.021348:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760739.021351:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.021354:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.021357:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760739.021360:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760739.021365:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760739.021372:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760739.021385:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760739.021466:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97480 - object 0x240000400:5 offset 24453120<>24457216 00000080:00200000:1.0:1789760739.058927:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5970, 5970] 00000080:00200000:1.0:1789760739.058940:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760739.058946:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760739.058956:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24457216 i_blocks 49159 00000020:00200000:1.0:1789760739.058959:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760739.058967:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760739.058990:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24457216, count: 4096 00000080:00200000:1.0:1789760739.058997:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24457216, bytes: 4096 00000080:00200000:1.0:1789760739.059004:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760739.059011:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760739.059013:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24457216, 24461312) 0 0 00000080:00200000:1.0:1789760739.059017:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760739.059029:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5971, 5971] 00020000:00200000:1.0:1789760739.059035:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24457216 chunk: [24457216, 24461312) 24461312, 4096 00020000:00200000:1.0:1789760739.059038:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760739.059052:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5971, 5971] 00000008:00010000:1.0:1789760739.059472:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760739.059477:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760739.059487:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24457216, 24461312) append 0 append lockpos 0 00000080:00200000:1.0:1789760739.059498:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24457216=0x1753000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760739.059541:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760739.059544:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24461312 00000020:00010000:1.0:1789760739.059551:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760739.059554:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.059556:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.059559:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760739.059561:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760739.059567:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760739.059572:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760739.059586:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760739.059659:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96680 - object 0x240000400:5 offset 24457216<>24461312 00000100:00100000:2.0:1789760739.102068:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e6cf7267-9b19-462a-a263-14115df15dab:2360:1876699512624640:192.168.203.156@tcp:400:kworker.0 00000100:00100000:2.0:1789760739.102190:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2360:1876699512624896:192.168.203.156@tcp:400:kworker.0 00000100:00100000:2.0:1789760739.102237:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760739.102244:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512624768:192.168.203.156@tcp:400:kworker.0 00000080:00200000:1.0:1789760739.106265:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5971, 5971] 00000080:00200000:1.0:1789760739.106279:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760739.106284:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760739.106294:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24461312 i_blocks 49159 00000020:00200000:1.0:1789760739.106298:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760739.106305:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760739.106329:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24461312, count: 4096 00000080:00200000:1.0:1789760739.106335:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24461312, bytes: 4096 00000080:00200000:1.0:1789760739.106342:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760739.106348:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760739.106350:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24461312, 24465408) 0 0 00000080:00200000:1.0:1789760739.106354:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760739.106366:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5972, 5972] 00020000:00200000:1.0:1789760739.106373:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24461312 chunk: [24461312, 24465408) 24465408, 4096 00020000:00200000:1.0:1789760739.106377:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760739.106390:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5972, 5972] 00000008:00010000:1.0:1789760739.106402:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760739.106407:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760739.106415:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24461312, 24465408) append 0 append lockpos 0 00000080:00200000:1.0:1789760739.106424:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24461312=0x1754000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760739.106475:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760739.106478:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24465408 00000020:00010000:1.0:1789760739.106486:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760739.106489:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.106492:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.106494:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760739.106497:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760739.106502:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760739.106507:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760739.106518:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760739.106590:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 24461312<>24465408 00000100:00100000:2.0:1789760739.106956:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2360:1876699512624896:192.168.203.156@tcp:400:kworker.0 00000100:00100000:2.0:1789760739.106977:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512624768:192.168.203.156@tcp:400:kworker.0 00000100:00100000:2.0:1789760739.107267:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e6cf7267-9b19-462a-a263-14115df15dab:2360:1876699512624640:192.168.203.156@tcp:400:kworker.0 00000080:00200000:1.0:1789760739.137273:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5972, 5972] 00000080:00200000:1.0:1789760739.137285:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760739.137290:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760739.137301:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24465408 i_blocks 49159 00000020:00200000:1.0:1789760739.137305:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760739.137312:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760739.137336:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24465408, count: 4096 00000080:00200000:1.0:1789760739.137342:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24465408, bytes: 4096 00000080:00200000:1.0:1789760739.137350:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760739.137356:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760739.137358:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24465408, 24469504) 0 0 00000080:00200000:1.0:1789760739.137362:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760739.137373:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5973, 5973] 00020000:00200000:1.0:1789760739.137379:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24465408 chunk: [24465408, 24469504) 24469504, 4096 00020000:00200000:1.0:1789760739.137383:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760739.137396:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5973, 5973] 00000008:00010000:1.0:1789760739.137408:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760739.137412:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760739.137421:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24465408, 24469504) append 0 append lockpos 0 00000080:00200000:1.0:1789760739.137432:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24465408=0x1755000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760739.137484:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760739.137487:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24469504 00000020:00010000:1.0:1789760739.137494:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760739.137497:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.137500:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.137503:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760739.137505:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760739.137511:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760739.137516:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760739.137530:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760739.137600:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 24465408<>24469504 00000080:00200000:1.0:1789760739.172609:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5973, 5973] 00000080:00200000:1.0:1789760739.172623:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760739.172629:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760739.172642:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24469504 i_blocks 49159 00000020:00200000:1.0:1789760739.172646:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760739.172655:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760739.172684:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24469504, count: 4096 00000080:00200000:1.0:1789760739.172691:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24469504, bytes: 4096 00000080:00200000:1.0:1789760739.172701:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760739.172710:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760739.172712:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24469504, 24473600) 0 0 00000080:00200000:1.0:1789760739.172716:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760739.172729:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5974, 5974] 00020000:00200000:1.0:1789760739.172736:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24469504 chunk: [24469504, 24473600) 24473600, 4096 00020000:00200000:1.0:1789760739.172740:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760739.172756:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5974, 5974] 00000008:00010000:1.0:1789760739.172769:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760739.172774:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760739.172785:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24469504, 24473600) append 0 append lockpos 0 00000080:00200000:1.0:1789760739.172795:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24469504=0x1756000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760739.172840:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760739.172842:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24473600 00000020:00010000:1.0:1789760739.172849:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760739.172852:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.172855:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.172858:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760739.172860:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760739.173100:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760739.173106:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760739.173124:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760739.173217:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94380 - object 0x240000400:5 offset 24469504<>24473600 00000080:00200000:1.0:1789760739.219485:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5974, 5974] 00000080:00200000:1.0:1789760739.219497:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760739.219502:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760739.219513:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24473600 i_blocks 49159 00000020:00200000:1.0:1789760739.219517:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760739.219524:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760739.219561:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24473600, count: 4096 00000080:00200000:1.0:1789760739.219568:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24473600, bytes: 4096 00000080:00200000:1.0:1789760739.219575:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760739.219582:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760739.219584:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24473600, 24477696) 0 0 00000080:00200000:1.0:1789760739.219587:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760739.219602:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5975, 5975] 00020000:00200000:1.0:1789760739.219609:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24473600 chunk: [24473600, 24477696) 24477696, 4096 00020000:00200000:1.0:1789760739.219613:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760739.219628:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5975, 5975] 00000008:00010000:1.0:1789760739.219641:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760739.219646:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760739.219655:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24473600, 24477696) append 0 append lockpos 0 00000080:00200000:1.0:1789760739.219666:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24473600=0x1757000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760739.219724:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760739.219727:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24477696 00000020:00010000:1.0:1789760739.219735:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760739.219739:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.219742:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.219746:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760739.219748:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760739.219755:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760739.219762:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760739.219775:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760739.219855:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 24473600<>24477696 00000080:00200000:1.0:1789760739.258893:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5975, 5975] 00000080:00200000:1.0:1789760739.258929:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760739.258935:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760739.258948:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24477696 i_blocks 49159 00000020:00200000:1.0:1789760739.258951:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760739.258968:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760739.258996:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24477696, count: 4096 00000080:00200000:1.0:1789760739.259002:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24477696, bytes: 4096 00000080:00200000:1.0:1789760739.259009:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760739.259016:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760739.259018:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24477696, 24481792) 0 0 00000080:00200000:1.0:1789760739.259022:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760739.259034:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5976, 5976] 00020000:00200000:1.0:1789760739.259041:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24477696 chunk: [24477696, 24481792) 24481792, 4096 00020000:00200000:1.0:1789760739.259045:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760739.259060:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5976, 5976] 00000008:00010000:1.0:1789760739.259074:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760739.259078:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760739.259088:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24477696, 24481792) append 0 append lockpos 0 00000080:00200000:1.0:1789760739.259100:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24477696=0x1758000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760739.259146:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760739.259149:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24481792 00000020:00010000:1.0:1789760739.259156:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760739.259160:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.259163:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.259166:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760739.259168:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760739.259174:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760739.259181:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760739.259194:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760739.259268:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 24477696<>24481792 00000080:00200000:1.0:1789760739.289257:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5976, 5976] 00000080:00200000:1.0:1789760739.289275:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760739.289280:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760739.289291:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24481792 i_blocks 49159 00000020:00200000:1.0:1789760739.289295:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760739.289302:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760739.289329:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24481792, count: 4096 00000080:00200000:1.0:1789760739.289335:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24481792, bytes: 4096 00000080:00200000:1.0:1789760739.289342:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760739.289349:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760739.289351:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24481792, 24485888) 0 0 00000080:00200000:1.0:1789760739.289354:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760739.289368:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5977, 5977] 00020000:00200000:1.0:1789760739.289374:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24481792 chunk: [24481792, 24485888) 24485888, 4096 00020000:00200000:1.0:1789760739.289378:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760739.289392:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5977, 5977] 00000008:00010000:1.0:1789760739.289406:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760739.289411:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760739.289419:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24481792, 24485888) append 0 append lockpos 0 00000080:00200000:1.0:1789760739.289429:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24481792=0x1759000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760739.289537:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760739.289540:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24485888 00000020:00010000:1.0:1789760739.289547:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760739.289550:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.289552:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.289555:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760739.289558:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760739.289564:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760739.289570:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760739.289585:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760739.289662:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94700 - object 0x240000400:5 offset 24481792<>24485888 00000080:00200000:1.0:1789760739.327747:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5977, 5977] 00000080:00200000:1.0:1789760739.327761:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760739.327766:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760739.327779:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24485888 i_blocks 49159 00000020:00200000:1.0:1789760739.327783:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760739.327945:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760739.327969:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24485888, count: 4096 00000080:00200000:1.0:1789760739.327974:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24485888, bytes: 4096 00000080:00200000:1.0:1789760739.327983:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760739.327990:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760739.327992:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24485888, 24489984) 0 0 00000080:00200000:1.0:1789760739.327996:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760739.328010:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5978, 5978] 00020000:00200000:1.0:1789760739.328016:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24485888 chunk: [24485888, 24489984) 24489984, 4096 00020000:00200000:1.0:1789760739.328020:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760739.328035:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5978, 5978] 00000008:00010000:1.0:1789760739.328048:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760739.328058:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760739.328071:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24485888, 24489984) append 0 append lockpos 0 00000080:00200000:1.0:1789760739.328081:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24485888=0x175a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760739.328125:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760739.328128:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24489984 00000020:00010000:1.0:1789760739.328135:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760739.328139:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.328141:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.328144:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760739.328147:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760739.328276:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760739.328356:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760739.328374:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760739.328477:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 24485888<>24489984 00000100:00100000:1.0:1789760739.328522:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760739.328538:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512625792:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760739.365854:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760739.365865:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760739.365872:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760739.365975:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760739.365997:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512625792:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760739.367961:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5978, 5978] 00000080:00200000:2.0:1789760739.367976:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.367980:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.367991:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24489984 i_blocks 49159 00000020:00200000:2.0:1789760739.367994:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.368005:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.368033:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24489984, count: 4096 00000080:00200000:2.0:1789760739.368040:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24489984, bytes: 4096 00000080:00200000:2.0:1789760739.368051:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.368059:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.368061:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24489984, 24494080) 0 0 00000080:00200000:2.0:1789760739.368065:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.368077:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5979, 5979] 00020000:00200000:2.0:1789760739.368084:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24489984 chunk: [24489984, 24494080) 24494080, 4096 00020000:00200000:2.0:1789760739.368088:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.368104:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5979, 5979] 00000008:00010000:2.0:1789760739.368117:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.368122:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760739.368132:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24489984, 24494080) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.368142:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24489984=0x175b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.368192:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.368195:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24494080 00000020:00010000:2.0:1789760739.368202:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.368205:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.368208:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.368210:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.368213:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.368221:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.368226:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.368246:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760739.368321:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 24489984<>24494080 00000100:00100000:2.0:1789760739.368370:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760739.368387:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512625920:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760739.404391:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973298 gen 3 00000008:00010000:2.0:1789760739.404459:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760739.404466:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760739.404474:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760739.404552:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760739.404577:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512625920:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760739.405160:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5979, 5979] 00000080:00200000:1.0:1789760739.405175:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760739.405179:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.405229:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24494080 i_blocks 49159 00000020:00200000:2.0:1789760739.405235:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.405242:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.405265:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24494080, count: 4096 00000080:00200000:2.0:1789760739.405270:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24494080, bytes: 4096 00000080:00200000:2.0:1789760739.405278:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.405284:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.405286:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24494080, 24498176) 0 0 00000080:00200000:2.0:1789760739.405289:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.405300:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5980, 5980] 00020000:00200000:2.0:1789760739.405305:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24494080 chunk: [24494080, 24498176) 24498176, 4096 00020000:00200000:2.0:1789760739.405308:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.405321:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5980, 5980] 00000008:00010000:2.0:1789760739.405334:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.405338:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760739.405345:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24494080, 24498176) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.405353:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24494080=0x175c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.405388:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.405390:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24498176 00000020:00010000:2.0:1789760739.405395:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.405398:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.405400:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.405403:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.405405:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.405410:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.405414:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.405426:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760739.405635:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 24494080<>24498176 00000100:00100000:3.0:1789760739.405719:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760739.405736:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512626048:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760739.438537:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973299 gen 3 00000008:00010000:2.0:1789760739.438612:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760739.438618:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760739.438624:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760739.438639:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000080:00200000:3.0:1789760739.438667:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5980, 5980] 00000080:00200000:3.0:1789760739.438680:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760739.438687:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760739.438699:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24498176 i_blocks 49159 00000020:00200000:3.0:1789760739.438703:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760739.438710:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000100:00100000:2.0:1789760739.438727:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512626048:192.168.203.156@tcp:4:dd.0 00000080:00200008:3.0:1789760739.438738:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24498176, count: 4096 00000080:00200000:3.0:1789760739.438744:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24498176, bytes: 4096 00000080:00200000:3.0:1789760739.438752:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760739.438759:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760739.438761:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24498176, 24502272) 0 0 00000080:00200000:3.0:1789760739.438764:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760739.438779:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5981, 5981] 00020000:00200000:3.0:1789760739.438786:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24498176 chunk: [24498176, 24502272) 24502272, 4096 00020000:00200000:3.0:1789760739.438790:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760739.438811:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5981, 5981] 00000008:00010000:3.0:1789760739.438881:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760739.438887:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760739.439108:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24498176, 24502272) append 0 append lockpos 0 00000080:00200000:3.0:1789760739.439121:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24498176=0x175d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760739.439165:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760739.439168:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24502272 00000020:00010000:3.0:1789760739.439174:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760739.439178:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.439180:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.439184:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760739.439186:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760739.439192:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760739.439197:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760739.439212:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760739.439366:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac380 - object 0x240000400:5 offset 24498176<>24502272 00000100:00100000:3.0:1789760739.439407:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760739.439420:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512626176:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760739.477426:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973300 gen 3 00000008:00010000:2.0:1789760739.477520:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760739.477531:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760739.477543:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760739.477563:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:2.0:1789760739.477586:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512626176:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760739.477762:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5981, 5981] 00000080:00200000:3.0:1789760739.477781:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760739.477790:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760739.477806:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24502272 i_blocks 49159 00000020:00200000:3.0:1789760739.477809:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760739.477818:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760739.477861:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24502272, count: 4096 00000080:00200000:3.0:1789760739.477868:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24502272, bytes: 4096 00000080:00200000:3.0:1789760739.477876:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760739.477883:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760739.477884:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24502272, 24506368) 0 0 00000080:00200000:3.0:1789760739.477888:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760739.477935:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5982, 5982] 00020000:00200000:3.0:1789760739.477952:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24502272 chunk: [24502272, 24506368) 24506368, 4096 00020000:00200000:3.0:1789760739.477955:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760739.477970:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5982, 5982] 00000008:00010000:3.0:1789760739.478633:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760739.478639:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760739.478652:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24502272, 24506368) append 0 append lockpos 0 00000080:00200000:3.0:1789760739.478663:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24502272=0x175e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760739.478714:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760739.478717:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24506368 00000020:00010000:3.0:1789760739.478723:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760739.478726:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.478729:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.478733:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760739.478735:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760739.478741:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760739.478746:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760739.478761:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760739.478849:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aea00 - object 0x240000400:5 offset 24502272<>24506368 00000100:00100000:3.0:1789760739.478895:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760739.479140:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512626304:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760739.503769:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973301 gen 3 00000008:00010000:2.0:1789760739.503806:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760739.503814:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760739.503824:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760739.503853:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760739.503875:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512626304:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760739.546364:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5983, 5983] 00000080:00200000:1.0:1789760739.546375:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760739.546379:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760739.546390:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24510464 i_blocks 49159 00000020:00200000:1.0:1789760739.546393:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760739.546402:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760739.546426:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24510464, count: 4096 00000080:00200000:1.0:1789760739.546431:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24510464, bytes: 4096 00000080:00200000:1.0:1789760739.546452:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760739.546460:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760739.546463:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24510464, 24514560) 0 0 00000080:00200000:1.0:1789760739.546467:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760739.546480:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5984, 5984] 00020000:00200000:1.0:1789760739.546487:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24510464 chunk: [24510464, 24514560) 24514560, 4096 00020000:00200000:1.0:1789760739.546490:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760739.546505:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5984, 5984] 00000008:00010000:1.0:1789760739.546519:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760739.546524:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760739.546532:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24510464, 24514560) append 0 append lockpos 0 00000080:00200000:1.0:1789760739.546543:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24510464=0x1760000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760739.546587:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760739.546590:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24514560 00000020:00010000:1.0:1789760739.546596:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760739.546600:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.546602:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760739.546605:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760739.546608:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760739.546613:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760739.546619:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760739.546633:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760739.546711:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 24510464<>24514560 00000100:00100000:1.0:1789760739.546749:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760739.546762:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512626560:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760739.586577:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973303 gen 3 00000008:00010000:1.0:1789760739.586614:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760739.586620:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760739.586629:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760739.586702:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760739.586725:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512626560:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760739.590621:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512626688:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760739.629468:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973304 gen 3 00000008:00010000:1.0:1789760739.629503:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760739.629510:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760739.629520:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760739.629544:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760739.629569:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512626688:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760739.630314:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512626816:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760739.667533:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760739.667543:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760739.667552:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760739.667574:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760739.667599:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512626816:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760739.668540:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760739.668556:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512626944:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760739.711817:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973306 gen 3 00000008:00010000:1.0:1789760739.712065:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760739.712071:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760739.712080:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760739.712101:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760739.712130:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512626944:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760739.713167:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512627072:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760739.756707:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973307 gen 3 00000008:00010000:1.0:1789760739.756746:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760739.756754:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760739.756763:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760739.757188:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760739.757213:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512627072:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760739.757821:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512627200:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760739.831344:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973308 gen 3 00000008:00010000:1.0:1789760739.831381:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760739.831386:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760739.831395:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760739.831472:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760739.831494:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512627200:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760739.831704:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5989, 5989] 00000080:00200000:2.0:1789760739.831715:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760739.831719:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760739.831730:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24535040 i_blocks 49159 00000020:00200000:2.0:1789760739.831733:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760739.831740:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760739.831764:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24535040, count: 4096 00000080:00200000:2.0:1789760739.831769:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24535040, bytes: 4096 00000080:00200000:2.0:1789760739.831776:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760739.831884:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760739.831887:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24535040, 24539136) 0 0 00000080:00200000:2.0:1789760739.831891:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760739.831990:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5990, 5990] 00020000:00200000:2.0:1789760739.831996:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24535040 chunk: [24535040, 24539136) 24539136, 4096 00020000:00200000:2.0:1789760739.832002:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760739.832018:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5990, 5990] 00000008:00010000:2.0:1789760739.832030:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760739.832035:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760739.832044:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24535040, 24539136) append 0 append lockpos 0 00000080:00200000:2.0:1789760739.832054:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24535040=0x1766000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760739.832200:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760739.832203:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24539136 00000020:00010000:2.0:1789760739.832211:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760739.832214:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.832216:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760739.832220:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760739.832223:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760739.832229:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760739.832234:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.832250:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760739.832323:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 24535040<>24539136 00000100:00100000:2.0:1789760739.832367:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760739.832381:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512627328:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760739.860669:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973309 gen 3 00000008:00010000:2.0:1789760739.860707:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760739.860714:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760739.860722:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760739.860735:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760739.860757:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512627328:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760739.861341:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5990, 5990] 00000080:00200000:3.0:1789760739.861356:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760739.861360:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760739.861372:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24539136 i_blocks 49159 00000020:00200000:3.0:1789760739.861376:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760739.861383:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760739.861410:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24539136, count: 4096 00000080:00200000:3.0:1789760739.861417:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24539136, bytes: 4096 00000080:00200000:3.0:1789760739.861427:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760739.861434:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760739.861445:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24539136, 24543232) 0 0 00000080:00200000:3.0:1789760739.861449:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760739.861461:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5991, 5991] 00020000:00200000:3.0:1789760739.861466:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24539136 chunk: [24539136, 24543232) 24543232, 4096 00020000:00200000:3.0:1789760739.861470:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760739.861485:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5991, 5991] 00000008:00010000:3.0:1789760739.861498:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760739.861503:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760739.861512:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24539136, 24543232) append 0 append lockpos 0 00000080:00200000:3.0:1789760739.861521:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24539136=0x1767000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760739.861559:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760739.861561:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24543232 00000020:00010000:3.0:1789760739.861568:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760739.861571:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.861574:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.861577:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760739.861579:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760739.861585:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760739.861591:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760739.861724:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:2.0:1789760739.861986:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 24539136<>24543232 00000100:00100000:2.0:1789760739.862031:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512627456:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760739.899739:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973310 gen 3 00000008:00010000:2.0:1789760739.899774:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760739.899780:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760739.899789:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760739.899856:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760739.899879:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512627456:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760739.900185:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5991, 5991] 00000080:00200000:3.0:1789760739.900197:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760739.900202:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760739.900213:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24543232 i_blocks 49159 00000020:00200000:3.0:1789760739.900216:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760739.900224:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760739.900248:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24543232, count: 4096 00000080:00200000:3.0:1789760739.900254:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24543232, bytes: 4096 00000080:00200000:3.0:1789760739.900261:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760739.900267:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760739.900268:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24543232, 24547328) 0 0 00000080:00200000:3.0:1789760739.900272:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760739.900285:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5992, 5992] 00020000:00200000:3.0:1789760739.900292:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24543232 chunk: [24543232, 24547328) 24547328, 4096 00020000:00200000:3.0:1789760739.900296:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760739.900309:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5992, 5992] 00000008:00010000:3.0:1789760739.900322:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760739.900326:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760739.900335:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24543232, 24547328) append 0 append lockpos 0 00000080:00200000:3.0:1789760739.900345:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24543232=0x1768000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760739.900388:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760739.900390:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24547328 00000020:00010000:3.0:1789760739.900396:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760739.900399:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.900402:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.900405:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760739.900408:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760739.900414:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760739.900419:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760739.900443:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760739.900516:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af100 - object 0x240000400:5 offset 24543232<>24547328 00000100:00100000:2.0:1789760739.900573:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512627584:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760739.942256:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760739.942267:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760739.942286:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760739.942366:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760739.942392:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512627584:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760739.942543:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5992, 5992] 00000080:00200000:3.0:1789760739.942557:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760739.942562:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760739.942573:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24547328 i_blocks 49159 00000020:00200000:3.0:1789760739.942576:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760739.942584:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760739.942667:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24547328, count: 4096 00000080:00200000:3.0:1789760739.942673:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24547328, bytes: 4096 00000080:00200000:3.0:1789760739.942684:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760739.942691:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760739.942693:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24547328, 24551424) 0 0 00000080:00200000:3.0:1789760739.942697:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760739.942709:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5993, 5993] 00020000:00200000:3.0:1789760739.942716:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24547328 chunk: [24547328, 24551424) 24551424, 4096 00020000:00200000:3.0:1789760739.942720:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760739.942734:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5993, 5993] 00000008:00010000:3.0:1789760739.942748:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760739.942752:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760739.942761:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24547328, 24551424) append 0 append lockpos 0 00000080:00200000:3.0:1789760739.942771:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24547328=0x1769000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760739.942812:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760739.942815:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24551424 00000020:00010000:3.0:1789760739.942821:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760739.942824:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.942827:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.942830:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760739.942833:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760739.942838:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760739.942844:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760739.942859:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760739.943344:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad180 - object 0x240000400:5 offset 24547328<>24551424 00000100:00100000:2.0:1789760739.943402:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512627712:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760739.972344:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973311 gen 3 00000100:00100000:2.0:1789760739.972375:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc0107ad180 x1876699512627712/t4294973312(4294973312) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760755 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760739.972410:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760739.972415:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760739.972423:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760739.972454:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760739.972473:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512627712:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760739.972585:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5993, 5993] 00000080:00200000:3.0:1789760739.972596:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760739.972602:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760739.972612:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24551424 i_blocks 49159 00000020:00200000:3.0:1789760739.972616:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760739.972623:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760739.972649:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24551424, count: 4096 00000080:00200000:3.0:1789760739.972656:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24551424, bytes: 4096 00000080:00200000:3.0:1789760739.972664:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760739.972671:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760739.972673:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24551424, 24555520) 0 0 00000080:00200000:3.0:1789760739.972677:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760739.972688:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5994, 5994] 00020000:00200000:3.0:1789760739.972695:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24551424 chunk: [24551424, 24555520) 24555520, 4096 00020000:00200000:3.0:1789760739.972698:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760739.972712:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5994, 5994] 00000008:00010000:3.0:1789760739.972725:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760739.972729:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760739.972743:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24551424, 24555520) append 0 append lockpos 0 00000080:00200000:3.0:1789760739.972754:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24551424=0x176a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760739.972795:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760739.972798:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24555520 00000020:00010000:3.0:1789760739.972804:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760739.972807:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.972810:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760739.972813:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760739.972816:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760739.972821:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760739.972893:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760739.972959:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760739.973034:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adf80 - object 0x240000400:5 offset 24551424<>24555520 00000100:00100000:3.0:1789760739.973069:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512627840:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760740.007122:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973312 gen 3 00000100:00100000:2.0:1789760740.007168:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc0107adf80 x1876699512627840/t4294973313(4294973313) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760755 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760740.007220:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760740.007225:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760740.007234:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760740.007254:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000080:00200000:3.0:1789760740.007336:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5994, 5994] 00000080:00200000:3.0:1789760740.007358:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760740.007366:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760740.007387:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24555520 i_blocks 49159 00000020:00200000:3.0:1789760740.007394:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760740.007407:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760740.007452:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24555520, count: 4096 00000080:00200000:3.0:1789760740.007465:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24555520, bytes: 4096 00000080:00200000:3.0:1789760740.007480:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760740.007487:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760740.007489:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24555520, 24559616) 0 0 00000080:00200000:3.0:1789760740.007493:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760740.007505:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5995, 5995] 00020000:00200000:3.0:1789760740.007511:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24555520 chunk: [24555520, 24559616) 24559616, 4096 00020000:00200000:3.0:1789760740.007515:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760740.007530:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5995, 5995] 00000008:00010000:3.0:1789760740.007544:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760740.007549:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760740.007558:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24555520, 24559616) append 0 append lockpos 0 00000080:00200000:3.0:1789760740.007570:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24555520=0x176b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760740.007618:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760740.007621:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24559616 00000020:00010000:3.0:1789760740.007628:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760740.007631:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.007634:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.007637:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760740.007646:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760740.007653:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760740.007658:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:2.0:1789760740.007768:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512627840:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760740.008885:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:2.0:1789760740.009150:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 24555520<>24559616 00000100:00100000:3.0:1789760740.009215:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512627968:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760740.040578:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973314 gen 3 00000008:00010000:2.0:1789760740.040621:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760740.040627:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760740.040635:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760740.040652:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760740.040677:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512627968:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760740.040683:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5995, 5995] 00000080:00200000:3.0:1789760740.040697:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760740.040702:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760740.040715:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24559616 i_blocks 49159 00000020:00200000:3.0:1789760740.040718:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760740.040727:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760740.040752:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24559616, count: 4096 00000080:00200000:3.0:1789760740.040758:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24559616, bytes: 4096 00000080:00200000:3.0:1789760740.040764:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760740.040770:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760740.040772:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24559616, 24563712) 0 0 00000080:00200000:3.0:1789760740.040776:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760740.040891:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5996, 5996] 00020000:00200000:3.0:1789760740.040982:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24559616 chunk: [24559616, 24563712) 24563712, 4096 00020000:00200000:3.0:1789760740.040987:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760740.041003:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5996, 5996] 00000008:00010000:3.0:1789760740.041016:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760740.041020:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760740.041030:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24559616, 24563712) append 0 append lockpos 0 00000080:00200000:3.0:1789760740.041040:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24559616=0x176c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760740.041081:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760740.041083:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24563712 00000020:00010000:3.0:1789760740.041089:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760740.041092:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.041094:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.041097:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760740.041100:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760740.041164:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760740.041171:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760740.041186:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760740.041259:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad880 - object 0x240000400:5 offset 24559616<>24563712 00000100:00100000:3.0:1789760740.041297:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760740.041310:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512628096:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760740.071959:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973315 gen 3 00000008:00010000:2.0:1789760740.071993:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760740.071998:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760740.072007:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760740.072068:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:2.0:1789760740.072091:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512628096:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760740.072259:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5996, 5996] 00000080:00200000:3.0:1789760740.072270:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760740.072275:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760740.072430:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24563712 i_blocks 49159 00000020:00200000:3.0:1789760740.072444:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760740.072453:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760740.072479:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24563712, count: 4096 00000080:00200000:3.0:1789760740.072485:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24563712, bytes: 4096 00000080:00200000:3.0:1789760740.072493:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760740.072565:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760740.072568:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24563712, 24567808) 0 0 00000080:00200000:3.0:1789760740.072572:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760740.072586:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5997, 5997] 00020000:00200000:3.0:1789760740.072593:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24563712 chunk: [24563712, 24567808) 24567808, 4096 00020000:00200000:3.0:1789760740.072596:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760740.072611:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5997, 5997] 00000008:00010000:3.0:1789760740.072625:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760740.072630:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760740.072639:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24563712, 24567808) append 0 append lockpos 0 00000080:00200000:3.0:1789760740.072650:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24563712=0x176d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760740.072695:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760740.072698:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24567808 00000020:00010000:3.0:1789760740.072704:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760740.072707:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.072709:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.072713:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760740.072715:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760740.072721:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760740.072726:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760740.072740:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760740.072814:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 24563712<>24567808 00000100:00100000:2.0:1789760740.072937:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512628224:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760740.117779:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760740.117791:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760740.117802:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760740.117823:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760740.117847:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512628224:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760740.117980:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5997, 5997] 00000080:00200000:3.0:1789760740.117994:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760740.117999:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760740.118010:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24567808 i_blocks 49159 00000020:00200000:3.0:1789760740.118014:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760740.118022:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760740.118046:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24567808, count: 4096 00000080:00200000:3.0:1789760740.118052:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24567808, bytes: 4096 00000080:00200000:3.0:1789760740.118062:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760740.118069:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760740.118071:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24567808, 24571904) 0 0 00000080:00200000:3.0:1789760740.118075:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760740.118088:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5998, 5998] 00020000:00200000:3.0:1789760740.118095:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24567808 chunk: [24567808, 24571904) 24571904, 4096 00020000:00200000:3.0:1789760740.118099:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760740.118233:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5998, 5998] 00000008:00010000:3.0:1789760740.118248:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760740.118253:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760740.118262:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24567808, 24571904) append 0 append lockpos 0 00000080:00200000:3.0:1789760740.118273:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24567808=0x176e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760740.118315:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760740.118318:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24571904 00000020:00010000:3.0:1789760740.118324:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760740.118327:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.118331:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.118334:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760740.118337:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760740.118343:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760740.118350:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760740.118364:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760740.118453:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae680 - object 0x240000400:5 offset 24567808<>24571904 00000100:00100000:2.0:1789760740.118558:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512628352:192.168.203.156@tcp:4:dd.0 00010000:00010000:1.0:1789760740.125624:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760740.158514:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973317 gen 3 00000008:00010000:2.0:1789760740.158561:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760740.158568:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760740.158577:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760740.158598:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760740.158621:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512628352:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760740.158711:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5998, 5998] 00000080:00200000:3.0:1789760740.158723:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760740.158728:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760740.158739:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24571904 i_blocks 49159 00000020:00200000:3.0:1789760740.158743:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760740.158750:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760740.158773:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24571904, count: 4096 00000080:00200000:3.0:1789760740.158779:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24571904, bytes: 4096 00000080:00200000:3.0:1789760740.158786:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760740.158793:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760740.158795:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24571904, 24576000) 0 0 00000080:00200000:3.0:1789760740.158799:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760740.158812:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [5999, 5999] 00020000:00200000:3.0:1789760740.158819:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24571904 chunk: [24571904, 24576000) 24576000, 4096 00020000:00200000:3.0:1789760740.158822:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760740.158837:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [5999, 5999] 00000008:00010000:3.0:1789760740.158849:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760740.158854:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760740.158863:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24571904, 24576000) append 0 append lockpos 0 00000080:00200000:3.0:1789760740.158874:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24571904=0x176f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760740.159071:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760740.159074:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24576000 00000020:00010000:3.0:1789760740.159082:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760740.159085:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.159087:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.159091:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760740.159093:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760740.159099:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760740.159105:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760740.159119:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760740.159190:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 24571904<>24576000 00000100:00100000:2.0:1789760740.159246:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512628480:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760740.194295:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760740.194305:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760740.194314:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760740.194331:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760740.194353:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512628480:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760740.194952:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [5999, 5999] 00000080:00200000:3.0:1789760740.194965:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760740.194969:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760740.194980:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24576000 i_blocks 49159 00000020:00200000:3.0:1789760740.194983:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760740.194990:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760740.195015:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24576000, count: 4096 00000080:00200000:3.0:1789760740.195020:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24576000, bytes: 4096 00000080:00200000:3.0:1789760740.195029:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760740.195035:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760740.195037:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24576000, 24580096) 0 0 00000080:00200000:3.0:1789760740.195040:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760740.195053:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6000, 6000] 00020000:00200000:3.0:1789760740.195060:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24576000 chunk: [24576000, 24580096) 24580096, 4096 00020000:00200000:3.0:1789760740.195064:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760740.195081:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6000, 6000] 00000008:00010000:3.0:1789760740.195096:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760740.195102:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760740.195111:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24576000, 24580096) append 0 append lockpos 0 00000080:00200000:3.0:1789760740.195125:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24576000=0x1770000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760740.195175:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760740.195178:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24580096 00000020:00010000:3.0:1789760740.195185:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760740.195188:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.195190:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.195194:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760740.195197:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760740.195204:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760740.195210:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760740.195226:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760740.195303:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af800 - object 0x240000400:5 offset 24576000<>24580096 00000100:00100000:2.0:1789760740.195489:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512628608:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760740.232174:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973319 gen 3 00000008:00010000:2.0:1789760740.232229:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760740.232235:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760740.232242:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760740.232254:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760740.232272:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512628608:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760740.232481:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6000, 6000] 00000080:00200000:3.0:1789760740.232493:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760740.232565:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760740.232578:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24580096 i_blocks 49159 00000020:00200000:3.0:1789760740.232581:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760740.232589:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760740.232616:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24580096, count: 4096 00000080:00200000:3.0:1789760740.232622:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24580096, bytes: 4096 00000080:00200000:3.0:1789760740.232629:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760740.232637:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760740.232639:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24580096, 24584192) 0 0 00000080:00200000:3.0:1789760740.232643:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760740.232655:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6001, 6001] 00020000:00200000:3.0:1789760740.232661:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24580096 chunk: [24580096, 24584192) 24584192, 4096 00020000:00200000:3.0:1789760740.232665:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760740.232680:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6001, 6001] 00000008:00010000:3.0:1789760740.232693:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760740.232698:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760740.232706:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24580096, 24584192) append 0 append lockpos 0 00000080:00200000:3.0:1789760740.232718:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24580096=0x1771000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760740.232760:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760740.232762:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24584192 00000020:00010000:3.0:1789760740.232769:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760740.232772:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.232775:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.232791:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760740.232795:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760740.232800:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760740.232806:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760740.232820:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760740.232897:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af800 - object 0x240000400:5 offset 24580096<>24584192 00000100:00100000:2.0:1789760740.233127:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512628736:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760740.272560:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973320 gen 3 00000008:00010000:2.0:1789760740.272599:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760740.272604:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760740.272613:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760740.272652:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000080:00200000:3.0:1789760740.272656:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6001, 6001] 00000080:00200000:3.0:1789760740.272669:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760740.272673:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:2.0:1789760740.272677:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512628736:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760740.272684:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24584192 i_blocks 49159 00000020:00200000:3.0:1789760740.272686:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760740.272694:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760740.272721:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24584192, count: 4096 00000080:00200000:3.0:1789760740.272726:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24584192, bytes: 4096 00000080:00200000:3.0:1789760740.272734:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760740.272741:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760740.272743:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24584192, 24588288) 0 0 00000080:00200000:3.0:1789760740.272747:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760740.272759:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6002, 6002] 00020000:00200000:3.0:1789760740.272766:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24584192 chunk: [24584192, 24588288) 24588288, 4096 00020000:00200000:3.0:1789760740.272769:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760740.272786:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6002, 6002] 00000008:00010000:3.0:1789760740.272892:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760740.272897:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760740.272926:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24584192, 24588288) append 0 append lockpos 0 00000080:00200000:3.0:1789760740.272937:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24584192=0x1772000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760740.272978:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760740.272981:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24588288 00000020:00010000:3.0:1789760740.272987:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760740.272990:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.272993:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.272996:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760740.272998:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760740.273003:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760740.273008:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760740.273022:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760740.273096:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 24584192<>24588288 00000100:00100000:2.0:1789760740.273320:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512628864:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760740.311212:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973321 gen 3 00000008:00010000:2.0:1789760740.311248:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760740.311254:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760740.311265:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760740.311333:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760740.311358:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512628864:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760740.311446:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6002, 6002] 00000080:00200000:3.0:1789760740.311460:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760740.311466:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760740.311477:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24588288 i_blocks 49159 00000020:00200000:3.0:1789760740.311481:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760740.311488:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760740.311514:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24588288, count: 4096 00000080:00200000:3.0:1789760740.311520:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24588288, bytes: 4096 00000080:00200000:3.0:1789760740.311529:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760740.311535:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760740.311537:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24588288, 24592384) 0 0 00000080:00200000:3.0:1789760740.311541:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760740.311554:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6003, 6003] 00020000:00200000:3.0:1789760740.311560:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24588288 chunk: [24588288, 24592384) 24592384, 4096 00020000:00200000:3.0:1789760740.311563:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760740.311577:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6003, 6003] 00000008:00010000:3.0:1789760740.311591:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760740.311595:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760740.311604:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24588288, 24592384) append 0 append lockpos 0 00000080:00200000:3.0:1789760740.311614:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24588288=0x1773000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760740.311658:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760740.311661:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24592384 00000020:00010000:3.0:1789760740.311668:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760740.311671:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.311673:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760740.311677:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760740.311680:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760740.311685:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760740.311691:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760740.311704:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760740.311784:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae680 - object 0x240000400:5 offset 24588288<>24592384 00000100:00100000:2.0:1789760740.312086:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512628992:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760740.355878:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973322 gen 3 00000008:00010000:2.0:1789760740.355938:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760740.355945:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760740.355954:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760740.355978:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760740.356000:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512628992:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760740.356577:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512629120:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760740.393688:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973323 gen 3 00000008:00010000:1.0:1789760740.393722:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760740.393728:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760740.393736:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760740.393763:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760740.393783:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512629120:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760740.394653:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512629248:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760740.423183:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760740.423194:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760740.423203:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760740.423224:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760740.423248:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512629248:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760740.423809:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512629376:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760740.456108:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973325 gen 3 00000008:00010000:1.0:1789760740.456153:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760740.456159:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760740.456167:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760740.456242:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760740.456265:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512629376:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760740.456406:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6006, 6006] 00000080:00200000:2.0:1789760740.456419:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760740.456424:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760740.456498:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24604672 i_blocks 49159 00000020:00200000:2.0:1789760740.456502:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760740.456510:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760740.456604:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24604672, count: 4096 00000080:00200000:2.0:1789760740.456611:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24604672, bytes: 4096 00000080:00200000:2.0:1789760740.456620:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760740.456627:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760740.456629:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24604672, 24608768) 0 0 00000080:00200000:2.0:1789760740.456633:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760740.456646:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6007, 6007] 00020000:00200000:2.0:1789760740.456653:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24604672 chunk: [24604672, 24608768) 24608768, 4096 00020000:00200000:2.0:1789760740.456656:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760740.456672:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6007, 6007] 00000008:00010000:2.0:1789760740.456685:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760740.456689:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760740.456698:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24604672, 24608768) append 0 append lockpos 0 00000080:00200000:2.0:1789760740.456708:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24604672=0x1777000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760740.456810:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760740.456813:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24608768 00000020:00010000:2.0:1789760740.456820:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760740.456875:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.456879:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760740.456882:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760740.456885:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760740.456892:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760740.456948:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760740.456964:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760740.457039:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 24604672<>24608768 00000100:00100000:3.0:1789760740.457103:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512629504:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760740.505619:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973326 gen 3 00000008:00010000:2.0:1789760740.505682:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760740.505691:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760740.505714:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760740.505742:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760740.505775:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512629504:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760740.506931:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512629632:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760740.546408:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973327 gen 3 00000008:00010000:1.0:1789760740.546460:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760740.546467:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760740.546478:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760740.546499:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760740.546522:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512629632:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760740.547104:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512629760:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760740.590324:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973328 gen 3 00000008:00010000:1.0:1789760740.590360:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760740.590366:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760740.590375:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760740.590396:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760740.590420:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512629760:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760740.591020:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512629888:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760740.647853:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973329 gen 3 00000008:00010000:1.0:1789760740.647889:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760740.647896:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760740.647938:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760740.647959:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760740.647983:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512629888:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760740.649177:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512630016:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760740.698146:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760740.698157:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760740.698167:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760740.698243:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760740.698267:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512630016:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760740.698774:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512630144:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760740.747679:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973331 gen 3 00000008:00010000:1.0:1789760740.747728:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760740.747737:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760740.747759:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760740.748367:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760740.748390:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512630144:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760740.749713:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512630272:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760740.822498:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973332 gen 3 00000008:00010000:1.0:1789760740.822543:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760740.822549:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760740.822559:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760740.822582:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760740.822607:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512630272:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760740.826402:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512630400:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760740.911035:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973333 gen 3 00000008:00010000:1.0:1789760740.911073:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760740.911080:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760740.911089:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760740.911112:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760740.911136:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512630400:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760740.912117:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760740.912133:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512630528:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760740.964276:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973334 gen 3 00000008:00010000:1.0:1789760740.964314:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760740.964321:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760740.964330:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760740.964355:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760740.964378:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512630528:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.016350:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6016, 6016] 00000080:00200000:2.0:1789760741.016363:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.016368:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.016381:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24645632 i_blocks 49159 00000020:00200000:2.0:1789760741.016385:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.016393:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.016417:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24645632, count: 4096 00000080:00200000:2.0:1789760741.016423:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24645632, bytes: 4096 00000080:00200000:2.0:1789760741.016441:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.016452:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.016454:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24645632, 24649728) 0 0 00000080:00200000:2.0:1789760741.016457:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.016470:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6017, 6017] 00020000:00200000:2.0:1789760741.016477:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24645632 chunk: [24645632, 24649728) 24649728, 4096 00020000:00200000:2.0:1789760741.016480:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.016495:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6017, 6017] 00000008:00010000:2.0:1789760741.016509:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.016514:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.016523:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24645632, 24649728) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.016534:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24645632=0x1781000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.016579:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.016581:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24649728 00000020:00010000:2.0:1789760741.016588:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.016591:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.016594:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.016597:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.016600:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.016607:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.016613:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.016631:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760741.016706:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 24645632<>24649728 00000100:00100000:3.0:1789760741.016770:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512630784:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760741.047814:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973336 gen 3 00000008:00010000:3.0:1789760741.047974:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760741.047981:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.047989:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.048044:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760741.048063:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512630784:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.048246:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6017, 6017] 00000080:00200000:2.0:1789760741.048258:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.048262:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.048273:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24649728 i_blocks 49159 00000020:00200000:2.0:1789760741.048276:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.048283:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.048306:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24649728, count: 4096 00000080:00200000:2.0:1789760741.048311:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24649728, bytes: 4096 00000080:00200000:2.0:1789760741.048319:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.048326:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.048328:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24649728, 24653824) 0 0 00000080:00200000:2.0:1789760741.048331:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.048342:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6018, 6018] 00020000:00200000:2.0:1789760741.048348:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24649728 chunk: [24649728, 24653824) 24653824, 4096 00020000:00200000:2.0:1789760741.048351:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.048365:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6018, 6018] 00000008:00010000:2.0:1789760741.048378:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.048383:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.048392:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24649728, 24653824) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.048402:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24649728=0x1782000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.048456:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.048459:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24653824 00000020:00010000:2.0:1789760741.048466:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.048469:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.048471:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.048474:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.048476:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.048483:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.048488:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.048501:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760741.048577:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 24649728<>24653824 00000100:00100000:3.0:1789760741.048642:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512630912:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760741.080424:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973337 gen 3 00000008:00010000:3.0:1789760741.080472:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760741.080480:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.080489:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.080508:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760741.080531:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512630912:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.080675:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6018, 6018] 00000080:00200000:2.0:1789760741.080688:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.080693:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.080705:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24653824 i_blocks 49159 00000020:00200000:2.0:1789760741.080708:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.080716:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.080741:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24653824, count: 4096 00000080:00200000:2.0:1789760741.080746:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24653824, bytes: 4096 00000080:00200000:2.0:1789760741.080754:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.080762:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.080764:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24653824, 24657920) 0 0 00000080:00200000:2.0:1789760741.080769:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.080784:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6019, 6019] 00020000:00200000:2.0:1789760741.080791:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24653824 chunk: [24653824, 24657920) 24657920, 4096 00020000:00200000:2.0:1789760741.080795:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.080809:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6019, 6019] 00000008:00010000:2.0:1789760741.080836:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.080842:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.080851:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24653824, 24657920) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.080997:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24653824=0x1783000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.081066:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.081069:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24657920 00000020:00010000:2.0:1789760741.081075:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.081078:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.081081:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.081084:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.081087:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.081094:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.081100:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.081114:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760741.081257:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 24653824<>24657920 00000100:00100000:3.0:1789760741.081661:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512631040:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760741.122309:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973338 gen 3 00000008:00010000:3.0:1789760741.122345:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760741.122352:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.122361:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.122376:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760741.122403:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512631040:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.122475:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6019, 6019] 00000080:00200000:2.0:1789760741.122489:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.122494:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.122504:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24657920 i_blocks 49159 00000020:00200000:2.0:1789760741.122508:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.122516:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.122541:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24657920, count: 4096 00000080:00200000:2.0:1789760741.122547:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24657920, bytes: 4096 00000080:00200000:2.0:1789760741.122555:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.122562:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.122564:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24657920, 24662016) 0 0 00000080:00200000:2.0:1789760741.122567:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.122578:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6020, 6020] 00020000:00200000:2.0:1789760741.122584:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24657920 chunk: [24657920, 24662016) 24662016, 4096 00020000:00200000:2.0:1789760741.122588:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.122602:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6020, 6020] 00000008:00010000:2.0:1789760741.122617:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.122621:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.122631:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24657920, 24662016) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.122641:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24657920=0x1784000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.122678:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.122680:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24662016 00000020:00010000:2.0:1789760741.122686:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.122690:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.122696:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.122700:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.122703:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.122710:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.122716:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760741.122733:0:54:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:3.0:1789760741.122824:0:54:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 24657920<>24662016 00000100:00100000:3.0:1789760741.122871:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512631168:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760741.151854:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760741.162386:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973339 gen 3 00000008:00010000:3.0:1789760741.162421:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760741.162437:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.162446:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.162499:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760741.162522:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512631168:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.162626:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6020, 6020] 00000080:00200000:2.0:1789760741.162638:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.162643:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.162654:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24662016 i_blocks 49159 00000020:00200000:2.0:1789760741.162657:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.162664:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.162687:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24662016, count: 4096 00000080:00200000:2.0:1789760741.162693:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24662016, bytes: 4096 00000080:00200000:2.0:1789760741.162700:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.162707:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.162709:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24662016, 24666112) 0 0 00000080:00200000:2.0:1789760741.162713:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.162724:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6021, 6021] 00020000:00200000:2.0:1789760741.162730:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24662016 chunk: [24662016, 24666112) 24666112, 4096 00020000:00200000:2.0:1789760741.162734:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.162748:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6021, 6021] 00000008:00010000:2.0:1789760741.162761:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.162765:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.162774:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24662016, 24666112) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.162785:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24662016=0x1785000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.163144:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.163147:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24666112 00000020:00010000:2.0:1789760741.163154:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.163158:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.163161:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.163164:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.163167:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.163174:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.163179:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.163193:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760741.163263:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 24662016<>24666112 00000100:00100000:3.0:1789760741.163312:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760741.163371:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512631296:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760741.198742:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973340 gen 3 00000008:00010000:3.0:1789760741.198780:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760741.198896:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.199065:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.199179:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760741.199204:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512631296:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.199274:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6021, 6021] 00000080:00200000:2.0:1789760741.199286:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.199292:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.199304:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24666112 i_blocks 49159 00000020:00200000:2.0:1789760741.199308:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.199316:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.199343:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24666112, count: 4096 00000080:00200000:2.0:1789760741.199349:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24666112, bytes: 4096 00000080:00200000:2.0:1789760741.199357:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.199364:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.199366:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24666112, 24670208) 0 0 00000080:00200000:2.0:1789760741.199370:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.199383:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6022, 6022] 00020000:00200000:2.0:1789760741.199390:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24666112 chunk: [24666112, 24670208) 24670208, 4096 00020000:00200000:2.0:1789760741.199393:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.199407:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6022, 6022] 00000008:00010000:2.0:1789760741.199495:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.199500:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.199511:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24666112, 24670208) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.199521:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24666112=0x1786000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.199567:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.199570:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24670208 00000020:00010000:2.0:1789760741.199577:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.199580:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.199582:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.199586:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.199589:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.199595:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.199601:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.199616:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760741.199693:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 24666112<>24670208 00000100:00100000:3.0:1789760741.199754:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512631424:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760741.241695:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760741.241705:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.241714:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.241730:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760741.241753:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512631424:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.241930:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6022, 6022] 00000080:00200000:2.0:1789760741.241942:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.241947:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.241958:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24670208 i_blocks 49159 00000020:00200000:2.0:1789760741.241961:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.241968:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.241990:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24670208, count: 4096 00000080:00200000:2.0:1789760741.241995:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24670208, bytes: 4096 00000080:00200000:2.0:1789760741.242003:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.242009:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.242011:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24670208, 24674304) 0 0 00000080:00200000:2.0:1789760741.242014:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.242026:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6023, 6023] 00020000:00200000:2.0:1789760741.242033:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24670208 chunk: [24670208, 24674304) 24674304, 4096 00020000:00200000:2.0:1789760741.242037:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.242051:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6023, 6023] 00000008:00010000:2.0:1789760741.242063:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.242067:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.242076:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24670208, 24674304) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.242085:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24670208=0x1787000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.242187:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.242190:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24674304 00000020:00010000:2.0:1789760741.242196:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.242199:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.242202:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.242205:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.242207:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.242213:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.242218:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.242231:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760741.242299:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 24670208<>24674304 00000100:00100000:2.0:1789760741.242337:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512631552:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760741.276497:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973342 gen 3 00000008:00010000:3.0:1789760741.276547:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760741.276554:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.276563:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.276582:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760741.276608:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512631552:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.276699:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6023, 6023] 00000080:00200000:2.0:1789760741.276715:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.276721:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.276738:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24674304 i_blocks 49159 00000020:00200000:2.0:1789760741.276742:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.276750:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.276782:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24674304, count: 4096 00000080:00200000:2.0:1789760741.276797:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24674304, bytes: 4096 00000080:00200000:2.0:1789760741.276805:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.276813:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.276815:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24674304, 24678400) 0 0 00000080:00200000:2.0:1789760741.276826:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.276841:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6024, 6024] 00020000:00200000:2.0:1789760741.276848:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24674304 chunk: [24674304, 24678400) 24678400, 4096 00020000:00200000:2.0:1789760741.276852:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.276871:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6024, 6024] 00000008:00010000:2.0:1789760741.276887:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.276891:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.276926:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24674304, 24678400) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.276938:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24674304=0x1788000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.276986:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.276988:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24678400 00000020:00010000:2.0:1789760741.276995:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.276999:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.277001:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.277005:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.277007:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.277013:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.277019:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.277034:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760741.277127:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 24674304<>24678400 00000100:00100000:3.0:1789760741.277443:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512631680:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760741.306628:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973343 gen 3 00000008:00010000:3.0:1789760741.306659:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760741.306665:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.306672:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.306686:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760741.306704:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512631680:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.306794:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6024, 6024] 00000080:00200000:2.0:1789760741.306807:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.306811:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.306824:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24678400 i_blocks 49159 00000020:00200000:2.0:1789760741.306828:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.306835:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.306862:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24678400, count: 4096 00000080:00200000:2.0:1789760741.306868:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24678400, bytes: 4096 00000080:00200000:2.0:1789760741.306875:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.306882:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.306884:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24678400, 24682496) 0 0 00000080:00200000:2.0:1789760741.306888:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.306925:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6025, 6025] 00020000:00200000:2.0:1789760741.306932:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24678400 chunk: [24678400, 24682496) 24682496, 4096 00020000:00200000:2.0:1789760741.306935:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.306951:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6025, 6025] 00000008:00010000:2.0:1789760741.306964:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.306969:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.306977:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24678400, 24682496) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.306988:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24678400=0x1789000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.307027:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.307030:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24682496 00000020:00010000:2.0:1789760741.307036:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.307039:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.307042:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.307045:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.307047:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.307053:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.307059:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.307073:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760741.307160:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 24678400<>24682496 00000100:00100000:2.0:1789760741.307200:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760741.307213:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512631808:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760741.340118:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760741.340129:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.340137:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.340245:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6025, 6025] 00000080:00200000:2.0:1789760741.340259:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.340263:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.340274:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24682496 i_blocks 49159 00000020:00200000:2.0:1789760741.340277:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.340284:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.340309:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24682496, count: 4096 00000080:00200000:2.0:1789760741.340316:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24682496, bytes: 4096 00000080:00200000:2.0:1789760741.340324:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.340331:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.340333:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24682496, 24686592) 0 0 00000080:00200000:2.0:1789760741.340337:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.340352:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6026, 6026] 00020000:00200000:2.0:1789760741.340359:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24682496 chunk: [24682496, 24686592) 24686592, 4096 00020000:00200000:2.0:1789760741.340363:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.340378:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6026, 6026] 00000008:00010000:2.0:1789760741.340393:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.340397:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.340406:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24682496, 24686592) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.340417:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24682496=0x178a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.340473:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.340476:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24686592 00000020:00010000:2.0:1789760741.340482:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.340485:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.340488:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.340491:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.340494:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.340500:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.340506:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.340523:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760741.340596:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 24682496<>24686592 00000100:00100000:2.0:1789760741.340625:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760741.340643:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512631936:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760741.341011:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760741.341029:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512631808:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760741.379010:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973345 gen 3 00000008:00010000:3.0:1789760741.379059:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760741.379065:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.379082:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.379098:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760741.379120:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512631936:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.379203:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6026, 6026] 00000080:00200000:2.0:1789760741.379216:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.379221:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.379231:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24686592 i_blocks 49159 00000020:00200000:2.0:1789760741.379237:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.379245:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.379282:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24686592, count: 4096 00000080:00200000:2.0:1789760741.379287:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24686592, bytes: 4096 00000080:00200000:2.0:1789760741.379294:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.379301:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.379304:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24686592, 24690688) 0 0 00000080:00200000:2.0:1789760741.379307:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.379320:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6027, 6027] 00020000:00200000:2.0:1789760741.379327:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24686592 chunk: [24686592, 24690688) 24690688, 4096 00020000:00200000:2.0:1789760741.379331:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.379346:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6027, 6027] 00000008:00010000:2.0:1789760741.379360:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.379365:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.379373:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24686592, 24690688) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.379383:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24686592=0x178b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.379445:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.379448:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24690688 00000020:00010000:2.0:1789760741.379454:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.379457:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.379461:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.379464:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.379466:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.379472:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.379477:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.379491:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760741.379785:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 24686592<>24690688 00000100:00100000:3.0:1789760741.380093:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512632064:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760741.429293:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973346 gen 3 00000008:00010000:3.0:1789760741.429331:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760741.429338:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.429346:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.429404:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760741.429438:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512632064:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.431047:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6027, 6027] 00000080:00200000:2.0:1789760741.431066:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.431071:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.431086:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24690688 i_blocks 49159 00000020:00200000:2.0:1789760741.431091:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.431103:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.431135:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24690688, count: 4096 00000080:00200000:2.0:1789760741.431142:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24690688, bytes: 4096 00000080:00200000:2.0:1789760741.431152:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.431160:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.431162:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24690688, 24694784) 0 0 00000080:00200000:2.0:1789760741.431166:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.431180:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6028, 6028] 00020000:00200000:2.0:1789760741.431187:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24690688 chunk: [24690688, 24694784) 24694784, 4096 00020000:00200000:2.0:1789760741.431191:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.431206:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6028, 6028] 00000008:00010000:2.0:1789760741.431221:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.431226:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.431235:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24690688, 24694784) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.431245:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24690688=0x178c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.431388:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.431392:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24694784 00000020:00010000:2.0:1789760741.431400:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.431403:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.431406:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.431408:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.431411:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.431418:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.431529:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.431555:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760741.431708:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 24690688<>24694784 00000100:00100000:3.0:1789760741.431792:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512632192:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760741.489569:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973347 gen 3 00000008:00010000:3.0:1789760741.489606:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760741.489613:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.489621:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.489640:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760741.489665:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512632192:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.489752:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6028, 6028] 00000080:00200000:2.0:1789760741.489765:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.489770:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.489782:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24694784 i_blocks 49159 00000020:00200000:2.0:1789760741.489786:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.489793:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.489829:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24694784, count: 4096 00000080:00200000:2.0:1789760741.489836:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24694784, bytes: 4096 00000080:00200000:2.0:1789760741.489843:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.489851:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.489853:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24694784, 24698880) 0 0 00000080:00200000:2.0:1789760741.489857:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.489871:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6029, 6029] 00020000:00200000:2.0:1789760741.489879:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24694784 chunk: [24694784, 24698880) 24698880, 4096 00020000:00200000:2.0:1789760741.489882:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.489897:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6029, 6029] 00000008:00010000:2.0:1789760741.489940:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.489944:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.489953:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24694784, 24698880) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.489964:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24694784=0x178d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.490007:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.490009:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24698880 00000020:00010000:2.0:1789760741.490015:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.490018:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.490021:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.490024:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.490027:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.490033:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.490039:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.490052:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760741.490140:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 24694784<>24698880 00000100:00100000:2.0:1789760741.490182:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512632320:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760741.525213:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973348 gen 3 00000008:00010000:3.0:1789760741.525246:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760741.525252:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.525259:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.525276:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760741.525300:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512632320:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.525377:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6029, 6029] 00000080:00200000:2.0:1789760741.525390:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.525394:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.525412:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24698880 i_blocks 49159 00000020:00200000:2.0:1789760741.525415:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.525430:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.525456:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24698880, count: 4096 00000080:00200000:2.0:1789760741.525463:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24698880, bytes: 4096 00000080:00200000:2.0:1789760741.525470:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.525478:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.525480:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24698880, 24702976) 0 0 00000080:00200000:2.0:1789760741.525483:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.525496:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6030, 6030] 00020000:00200000:2.0:1789760741.525570:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24698880 chunk: [24698880, 24702976) 24702976, 4096 00020000:00200000:2.0:1789760741.525576:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.525590:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6030, 6030] 00000008:00010000:2.0:1789760741.525601:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.525605:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.525613:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24698880, 24702976) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.525622:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24698880=0x178e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.525657:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.525659:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24702976 00000020:00010000:2.0:1789760741.525663:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.525665:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.525667:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.525669:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.525671:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.525675:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.525680:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.525691:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760741.525754:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 24698880<>24702976 00000100:00100000:3.0:1789760741.525932:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760741.525946:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512632448:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760741.575075:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973349 gen 3 00000008:00010000:3.0:1789760741.575113:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760741.575119:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.575128:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.575185:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760741.575208:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512632448:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.575323:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6030, 6030] 00000080:00200000:2.0:1789760741.575335:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.575342:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.575354:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24702976 i_blocks 49159 00000020:00200000:2.0:1789760741.575357:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.575365:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.575390:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24702976, count: 4096 00000080:00200000:2.0:1789760741.575397:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24702976, bytes: 4096 00000080:00200000:2.0:1789760741.575405:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.575412:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.575414:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24702976, 24707072) 0 0 00000080:00200000:2.0:1789760741.575418:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.575438:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6031, 6031] 00020000:00200000:2.0:1789760741.575444:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24702976 chunk: [24702976, 24707072) 24707072, 4096 00020000:00200000:2.0:1789760741.575448:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.575463:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6031, 6031] 00000008:00010000:2.0:1789760741.575476:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.575481:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.575489:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24702976, 24707072) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.575500:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24702976=0x178f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.575541:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.575544:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24707072 00000020:00010000:2.0:1789760741.575550:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.575552:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.575554:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.575556:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.575558:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.575562:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.575566:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.575577:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000008:00100000:2.0:1789760741.575645:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 24702976<>24707072 00000100:00100000:3.0:1789760741.575707:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512632576:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760741.619883:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973350 gen 3 00000008:00010000:3.0:1789760741.619944:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760741.619951:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.619960:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.619977:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760741.620000:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512632576:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.620230:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6031, 6031] 00000080:00200000:2.0:1789760741.620243:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.620248:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.620259:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24707072 i_blocks 49159 00000020:00200000:2.0:1789760741.620262:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.620270:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.620295:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24707072, count: 4096 00000080:00200000:2.0:1789760741.620302:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24707072, bytes: 4096 00000080:00200000:2.0:1789760741.620310:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.620318:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.620320:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24707072, 24711168) 0 0 00000080:00200000:2.0:1789760741.620324:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.620337:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6032, 6032] 00020000:00200000:2.0:1789760741.620343:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24707072 chunk: [24707072, 24711168) 24711168, 4096 00020000:00200000:2.0:1789760741.620347:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.620360:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6032, 6032] 00000008:00010000:2.0:1789760741.620374:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.620378:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.620387:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24707072, 24711168) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.620397:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24707072=0x1790000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.620514:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.620517:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24711168 00000020:00010000:2.0:1789760741.620523:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.620527:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.620529:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.620532:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.620535:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.620541:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.620546:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.620562:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760741.620638:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 24707072<>24711168 00000100:00100000:2.0:1789760741.620685:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512632704:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760741.662680:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760741.662689:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.662698:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.662717:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760741.662738:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512632704:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.662953:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6032, 6032] 00000080:00200000:2.0:1789760741.663283:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.663289:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.663300:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24711168 i_blocks 49159 00000020:00200000:2.0:1789760741.663303:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.663312:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.663340:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24711168, count: 4096 00000080:00200000:2.0:1789760741.663345:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24711168, bytes: 4096 00000080:00200000:2.0:1789760741.663354:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.663361:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.663363:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24711168, 24715264) 0 0 00000080:00200000:2.0:1789760741.663367:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.663393:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6033, 6033] 00020000:00200000:2.0:1789760741.663401:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24711168 chunk: [24711168, 24715264) 24715264, 4096 00020000:00200000:2.0:1789760741.663405:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.663432:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6033, 6033] 00000008:00010000:2.0:1789760741.663447:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.663452:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.663460:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24711168, 24715264) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.663471:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24711168=0x1791000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.663514:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.663517:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24715264 00000020:00010000:2.0:1789760741.663523:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.663527:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.663530:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.663533:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.663536:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.663542:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.663548:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.663567:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760741.663649:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 24711168<>24715264 00000100:00100000:3.0:1789760741.663890:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512632832:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760741.692540:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973351 gen 3 00000100:00100000:3.0:1789760741.692568:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368ca80 x1876699512632832/t4294973352(4294973352) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760757 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760741.692603:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760741.692608:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.692616:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.692684:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760741.692705:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512632832:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.693118:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6033, 6033] 00000080:00200000:2.0:1789760741.693129:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.693134:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.693147:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24715264 i_blocks 49159 00000020:00200000:2.0:1789760741.693150:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.693158:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.693181:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24715264, count: 4096 00000080:00200000:2.0:1789760741.693187:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24715264, bytes: 4096 00000080:00200000:2.0:1789760741.693196:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.693203:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.693205:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24715264, 24719360) 0 0 00000080:00200000:2.0:1789760741.693209:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.693220:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6034, 6034] 00020000:00200000:2.0:1789760741.693226:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24715264 chunk: [24715264, 24719360) 24719360, 4096 00020000:00200000:2.0:1789760741.693230:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.693243:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6034, 6034] 00000008:00010000:2.0:1789760741.693254:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.693259:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.693267:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24715264, 24719360) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.693277:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24715264=0x1792000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.693316:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.693319:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24719360 00000020:00010000:2.0:1789760741.693324:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.693327:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.693330:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.693333:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.693336:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.693343:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.693348:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.693365:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000008:00100000:2.0:1789760741.693451:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 24715264<>24719360 00000100:00100000:3.0:1789760741.693507:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760741.695327:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512632960:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760741.733763:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973353 gen 3 00000008:00010000:3.0:1789760741.733807:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760741.733814:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.733821:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.733876:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760741.733936:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512632960:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.734165:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6034, 6034] 00000080:00200000:2.0:1789760741.734178:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.734183:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.734195:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24719360 i_blocks 49159 00000020:00200000:2.0:1789760741.734198:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.734207:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.734232:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24719360, count: 4096 00000080:00200000:2.0:1789760741.734238:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24719360, bytes: 4096 00000080:00200000:2.0:1789760741.734245:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.734252:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.734254:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24719360, 24723456) 0 0 00000080:00200000:2.0:1789760741.734259:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.734272:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6035, 6035] 00020000:00200000:2.0:1789760741.734278:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24719360 chunk: [24719360, 24723456) 24723456, 4096 00020000:00200000:2.0:1789760741.734282:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.735135:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6035, 6035] 00000008:00010000:2.0:1789760741.735154:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.735159:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.735169:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24719360, 24723456) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.735179:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24719360=0x1793000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.735227:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.735230:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24723456 00000020:00010000:2.0:1789760741.735237:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.735241:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.735243:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.735247:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.735249:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.735256:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.735262:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.735285:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000008:00100000:2.0:1789760741.735382:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 24719360<>24723456 00000100:00100000:2.0:1789760741.735432:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760741.735451:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512633088:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760741.773217:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973354 gen 3 00000008:00010000:3.0:1789760741.773250:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760741.773256:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.773264:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.773277:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760741.773299:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512633088:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.773305:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6035, 6035] 00000080:00200000:2.0:1789760741.773320:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.773327:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.773338:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24723456 i_blocks 49159 00000020:00200000:2.0:1789760741.773342:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.773352:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.773378:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24723456, count: 4096 00000080:00200000:2.0:1789760741.773384:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24723456, bytes: 4096 00000080:00200000:2.0:1789760741.773391:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.773398:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.773412:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24723456, 24727552) 0 0 00000080:00200000:2.0:1789760741.773417:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.773441:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6036, 6036] 00020000:00200000:2.0:1789760741.773448:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24723456 chunk: [24723456, 24727552) 24727552, 4096 00020000:00200000:2.0:1789760741.773452:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.773466:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6036, 6036] 00000008:00010000:2.0:1789760741.773479:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.773483:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.773493:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24723456, 24727552) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.773503:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24723456=0x1794000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.773545:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.773548:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24727552 00000020:00010000:2.0:1789760741.773554:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.773557:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.773559:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.773562:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.773565:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.773571:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.773576:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.773590:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760741.773662:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 24723456<>24727552 00000100:00100000:3.0:1789760741.773920:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760741.773934:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512633216:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760741.806992:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973355 gen 3 00000008:00010000:3.0:1789760741.807027:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760741.807033:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.807041:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.807059:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760741.807081:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512633216:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.807177:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6036, 6036] 00000080:00200000:2.0:1789760741.807190:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.807196:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.807207:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24727552 i_blocks 49159 00000020:00200000:2.0:1789760741.807211:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.807218:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.807243:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24727552, count: 4096 00000080:00200000:2.0:1789760741.807249:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24727552, bytes: 4096 00000080:00200000:2.0:1789760741.807257:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.807264:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.807266:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24727552, 24731648) 0 0 00000080:00200000:2.0:1789760741.807269:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.807283:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6037, 6037] 00020000:00200000:2.0:1789760741.807290:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24727552 chunk: [24727552, 24731648) 24731648, 4096 00020000:00200000:2.0:1789760741.807294:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.807308:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6037, 6037] 00000008:00010000:2.0:1789760741.807322:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.807326:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.807335:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24727552, 24731648) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.807345:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24727552=0x1795000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.807388:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.807390:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24731648 00000020:00010000:2.0:1789760741.807396:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.807399:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.807402:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.807405:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.807407:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.807413:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.807419:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.808008:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760741.808084:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 24727552<>24731648 00000100:00100000:3.0:1789760741.808149:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512633344:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760741.869617:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973356 gen 3 00000008:00010000:3.0:1789760741.869656:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760741.869662:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.869671:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.869689:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760741.869712:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512633344:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.869803:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6037, 6037] 00000080:00200000:2.0:1789760741.869816:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.869821:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.869832:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24731648 i_blocks 49159 00000020:00200000:2.0:1789760741.869835:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.869843:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.869869:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24731648, count: 4096 00000080:00200000:2.0:1789760741.869875:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24731648, bytes: 4096 00000080:00200000:2.0:1789760741.869882:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.869889:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.869891:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24731648, 24735744) 0 0 00000080:00200000:2.0:1789760741.869895:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.869923:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6038, 6038] 00020000:00200000:2.0:1789760741.869930:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24731648 chunk: [24731648, 24735744) 24735744, 4096 00020000:00200000:2.0:1789760741.869933:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.869948:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6038, 6038] 00000008:00010000:2.0:1789760741.869960:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.869964:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.869973:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24731648, 24735744) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.869983:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24731648=0x1796000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.870025:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.870028:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24735744 00000020:00010000:2.0:1789760741.870034:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.870037:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.870040:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.870042:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.870045:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.870051:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.870057:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.870070:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000008:00100000:2.0:1789760741.870144:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 24731648<>24735744 00000100:00100000:3.0:1789760741.870192:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760741.870888:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512633472:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760741.905761:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973357 gen 3 00000008:00010000:3.0:1789760741.905791:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760741.905878:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.905887:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.905962:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760741.905995:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512633472:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.906167:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6038, 6038] 00000080:00200000:2.0:1789760741.906180:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.906186:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.906198:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24735744 i_blocks 49159 00000020:00200000:2.0:1789760741.906201:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.906208:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.906234:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24735744, count: 4096 00000080:00200000:2.0:1789760741.906240:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24735744, bytes: 4096 00000080:00200000:2.0:1789760741.906247:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.906254:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.906256:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24735744, 24739840) 0 0 00000080:00200000:2.0:1789760741.906260:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.906272:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6039, 6039] 00020000:00200000:2.0:1789760741.906279:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24735744 chunk: [24735744, 24739840) 24739840, 4096 00020000:00200000:2.0:1789760741.906283:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.906366:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6039, 6039] 00000008:00010000:2.0:1789760741.906382:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.906387:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.906396:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24735744, 24739840) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.906406:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24735744=0x1797000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.906464:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.906467:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24739840 00000020:00010000:2.0:1789760741.906474:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.906477:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.906481:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.906484:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.906488:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.906494:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.906500:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.906514:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760741.906665:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 24735744<>24739840 00000100:00100000:3.0:1789760741.906722:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512633600:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760741.951965:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760741.951977:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.951987:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.952005:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760741.952030:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512633600:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.953342:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6039, 6039] 00000080:00200000:2.0:1789760741.953363:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.953370:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.953384:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24739840 i_blocks 49159 00000020:00200000:2.0:1789760741.953388:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.953397:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.953433:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24739840, count: 4096 00000080:00200000:2.0:1789760741.953440:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24739840, bytes: 4096 00000080:00200000:2.0:1789760741.953449:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.953457:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.953459:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24739840, 24743936) 0 0 00000080:00200000:2.0:1789760741.953463:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.953479:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6040, 6040] 00020000:00200000:2.0:1789760741.953486:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24739840 chunk: [24739840, 24743936) 24743936, 4096 00020000:00200000:2.0:1789760741.953490:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.953505:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6040, 6040] 00000008:00010000:2.0:1789760741.953520:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.953525:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.953535:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24739840, 24743936) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.953548:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24739840=0x1798000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.953599:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.953602:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24743936 00000020:00010000:2.0:1789760741.953608:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.953612:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.953614:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.953618:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.953621:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.953628:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.953634:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.953651:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760741.953735:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 24739840<>24743936 00000100:00100000:2.0:1789760741.953788:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512633728:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760741.993876:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973359 gen 3 00000008:00010000:3.0:1789760741.993950:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760741.993957:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760741.993965:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760741.993981:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760741.994001:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512633728:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760741.994430:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6040, 6040] 00000080:00200000:2.0:1789760741.994448:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760741.994453:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760741.994465:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24743936 i_blocks 49159 00000020:00200000:2.0:1789760741.994469:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760741.994477:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760741.994514:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24743936, count: 4096 00000080:00200000:2.0:1789760741.994521:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24743936, bytes: 4096 00000080:00200000:2.0:1789760741.994528:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760741.994535:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760741.994537:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24743936, 24748032) 0 0 00000080:00200000:2.0:1789760741.994540:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760741.994553:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6041, 6041] 00020000:00200000:2.0:1789760741.994560:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24743936 chunk: [24743936, 24748032) 24748032, 4096 00020000:00200000:2.0:1789760741.994564:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760741.994579:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6041, 6041] 00000008:00010000:2.0:1789760741.994592:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760741.994597:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760741.994606:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24743936, 24748032) append 0 append lockpos 0 00000080:00200000:2.0:1789760741.994616:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24743936=0x1799000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760741.994658:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760741.994661:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24748032 00000020:00010000:2.0:1789760741.994667:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760741.994670:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.994672:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760741.994675:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760741.994678:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760741.994684:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760741.994689:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760741.994705:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760741.994780:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 24743936<>24748032 00000100:00100000:3.0:1789760741.994879:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512633856:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760742.057283:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973360 gen 3 00000008:00010000:3.0:1789760742.057322:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760742.057329:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.057338:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.057357:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760742.057381:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512633856:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760742.057485:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6041, 6041] 00000080:00200000:2.0:1789760742.057500:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.057505:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.057517:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24748032 i_blocks 49159 00000020:00200000:2.0:1789760742.057521:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.057529:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.057556:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24748032, count: 4096 00000080:00200000:2.0:1789760742.057562:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24748032, bytes: 4096 00000080:00200000:2.0:1789760742.057570:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.057578:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.057580:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24748032, 24752128) 0 0 00000080:00200000:2.0:1789760742.057584:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.057597:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6042, 6042] 00020000:00200000:2.0:1789760742.057604:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24748032 chunk: [24748032, 24752128) 24752128, 4096 00020000:00200000:2.0:1789760742.057607:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.057623:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6042, 6042] 00000008:00010000:2.0:1789760742.057636:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.057641:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760742.057650:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24748032, 24752128) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.057662:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24748032=0x179a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.057705:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.057707:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24752128 00000020:00010000:2.0:1789760742.057714:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.057717:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.057720:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.057723:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.057725:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.057732:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.057737:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.057749:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760742.057823:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 24748032<>24752128 00000100:00100000:3.0:1789760742.057880:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512633984:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760742.095267:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973361 gen 3 00000008:00010000:3.0:1789760742.095304:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760742.095310:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.095319:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.095340:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760742.095365:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512633984:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760742.095748:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6042, 6042] 00000080:00200000:2.0:1789760742.095763:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.095769:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.095783:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24752128 i_blocks 49159 00000020:00200000:2.0:1789760742.095786:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.095805:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.095833:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24752128, count: 4096 00000080:00200000:2.0:1789760742.095839:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24752128, bytes: 4096 00000080:00200000:2.0:1789760742.095848:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.095855:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.095857:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24752128, 24756224) 0 0 00000080:00200000:2.0:1789760742.095861:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.095874:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6043, 6043] 00020000:00200000:2.0:1789760742.095880:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24752128 chunk: [24752128, 24756224) 24756224, 4096 00020000:00200000:2.0:1789760742.095884:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.095926:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6043, 6043] 00000008:00010000:2.0:1789760742.095941:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.095945:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760742.095955:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24752128, 24756224) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.095966:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24752128=0x179b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.096010:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.096012:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24756224 00000020:00010000:2.0:1789760742.096019:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.096022:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.096024:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.096027:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.096030:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.096037:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.096043:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.096056:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760742.096138:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 24752128<>24756224 00000100:00100000:3.0:1789760742.096563:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512634112:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760742.144251:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973362 gen 3 00000008:00010000:3.0:1789760742.144286:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760742.144292:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.144300:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.144321:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760742.144345:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512634112:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760742.144642:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6043, 6043] 00000080:00200000:2.0:1789760742.144656:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.144661:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.144672:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24756224 i_blocks 49159 00000020:00200000:2.0:1789760742.144676:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.144685:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.144710:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24756224, count: 4096 00000080:00200000:2.0:1789760742.144717:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24756224, bytes: 4096 00000080:00200000:2.0:1789760742.144725:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.144733:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.144735:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24756224, 24760320) 0 0 00000080:00200000:2.0:1789760742.144739:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.144752:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6044, 6044] 00020000:00200000:2.0:1789760742.144759:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24756224 chunk: [24756224, 24760320) 24760320, 4096 00020000:00200000:2.0:1789760742.144763:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.144778:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6044, 6044] 00000008:00010000:2.0:1789760742.144791:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.144796:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760742.144806:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24756224, 24760320) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.145191:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24756224=0x179c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.145239:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.145242:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24760320 00000020:00010000:2.0:1789760742.145249:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.145252:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.145255:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.145258:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.145260:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.145266:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.145272:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.145289:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760742.145365:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 24756224<>24760320 00000100:00100000:3.0:1789760742.145433:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760742.145449:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512634240:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760742.191827:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973363 gen 3 00000008:00010000:3.0:1789760742.191861:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760742.191867:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.191875:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.191895:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760742.192016:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512634240:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760742.192207:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6044, 6044] 00000080:00200000:2.0:1789760742.192219:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.192224:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.192235:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24760320 i_blocks 49159 00000020:00200000:2.0:1789760742.192238:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.192248:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.192275:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24760320, count: 4096 00000080:00200000:2.0:1789760742.192281:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24760320, bytes: 4096 00000080:00200000:2.0:1789760742.192289:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.192296:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.192298:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24760320, 24764416) 0 0 00000080:00200000:2.0:1789760742.192302:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.192315:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6045, 6045] 00020000:00200000:2.0:1789760742.192324:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24760320 chunk: [24760320, 24764416) 24764416, 4096 00020000:00200000:2.0:1789760742.192327:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.192343:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6045, 6045] 00000008:00010000:2.0:1789760742.192356:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.192360:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760742.192370:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24760320, 24764416) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.192380:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24760320=0x179d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.192434:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.192437:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24764416 00000020:00010000:2.0:1789760742.192443:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.192447:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.192450:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.192514:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.192519:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.192527:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.192532:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.192546:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760742.192625:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 24760320<>24764416 00000100:00100000:3.0:1789760742.193557:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512634368:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760742.223185:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973364 gen 3 00000008:00010000:3.0:1789760742.223221:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760742.223227:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.223235:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.223254:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760742.223283:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512634368:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760742.223310:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6045, 6045] 00000080:00200000:2.0:1789760742.223327:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.223332:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.223346:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24764416 i_blocks 49159 00000020:00200000:2.0:1789760742.223350:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.223359:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.223386:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24764416, count: 4096 00000080:00200000:2.0:1789760742.223391:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24764416, bytes: 4096 00000080:00200000:2.0:1789760742.223399:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.223406:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.223408:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24764416, 24768512) 0 0 00000080:00200000:2.0:1789760742.223412:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.223440:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6046, 6046] 00020000:00200000:2.0:1789760742.223446:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24764416 chunk: [24764416, 24768512) 24768512, 4096 00020000:00200000:2.0:1789760742.223450:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.223465:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6046, 6046] 00000008:00010000:2.0:1789760742.223480:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.223484:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760742.223492:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24764416, 24768512) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.223503:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24764416=0x179e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.223756:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.223760:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24768512 00000020:00010000:2.0:1789760742.223767:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.223771:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.223773:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.223777:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.223780:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.223787:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.223792:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.223807:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760742.223882:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 24764416<>24768512 00000100:00100000:3.0:1789760742.223953:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512634496:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760742.268629:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973365 gen 3 00000008:00010000:3.0:1789760742.268664:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760742.268670:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.268678:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.268697:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760742.268718:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512634496:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760742.268849:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6046, 6046] 00000080:00200000:2.0:1789760742.268862:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.268867:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.268882:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24768512 i_blocks 49159 00000020:00200000:2.0:1789760742.268885:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.268893:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.268944:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24768512, count: 4096 00000080:00200000:2.0:1789760742.268951:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24768512, bytes: 4096 00000080:00200000:2.0:1789760742.268958:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.268965:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.268967:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24768512, 24772608) 0 0 00000080:00200000:2.0:1789760742.268971:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.268984:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6047, 6047] 00020000:00200000:2.0:1789760742.269181:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24768512 chunk: [24768512, 24772608) 24772608, 4096 00020000:00200000:2.0:1789760742.269186:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.269201:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6047, 6047] 00000008:00010000:2.0:1789760742.269215:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.269220:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760742.269230:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24768512, 24772608) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.269240:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24768512=0x179f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.269282:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.269284:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24772608 00000020:00010000:2.0:1789760742.269290:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.269294:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.269297:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.269300:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.269303:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.269310:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.269316:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.269330:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760742.269399:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 24768512<>24772608 00000100:00100000:3.0:1789760742.269492:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512634624:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760742.309499:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973366 gen 3 00000008:00010000:3.0:1789760742.309532:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760742.309537:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.309543:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.309593:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760742.309614:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512634624:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760742.309807:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6047, 6047] 00000080:00200000:2.0:1789760742.309827:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.309831:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.309843:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24772608 i_blocks 49159 00000020:00200000:2.0:1789760742.309846:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.309854:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.309879:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24772608, count: 4096 00000080:00200000:2.0:1789760742.309885:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24772608, bytes: 4096 00000080:00200000:2.0:1789760742.309892:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.309921:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.309923:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24772608, 24776704) 0 0 00000080:00200000:2.0:1789760742.309927:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.309939:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6048, 6048] 00020000:00200000:2.0:1789760742.309946:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24772608 chunk: [24772608, 24776704) 24776704, 4096 00020000:00200000:2.0:1789760742.309950:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.309964:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6048, 6048] 00000008:00010000:2.0:1789760742.309978:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.309984:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760742.309993:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24772608, 24776704) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.310003:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24772608=0x17a0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.310046:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.310049:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24776704 00000020:00010000:2.0:1789760742.310055:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.310058:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.310061:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.310064:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.310067:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.310073:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.310078:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.310101:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760742.310180:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 24772608<>24776704 00000100:00100000:3.0:1789760742.310257:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760742.310271:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512634752:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760742.348974:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973367 gen 3 00000008:00010000:3.0:1789760742.349013:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760742.349020:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.349029:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.349082:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760742.349116:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512634752:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760742.349233:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6048, 6048] 00000080:00200000:2.0:1789760742.349247:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.349252:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.349264:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24776704 i_blocks 49159 00000020:00200000:2.0:1789760742.349268:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.349277:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.349303:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24776704, count: 4096 00000080:00200000:2.0:1789760742.349309:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24776704, bytes: 4096 00000080:00200000:2.0:1789760742.349316:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.349324:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.349326:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24776704, 24780800) 0 0 00000080:00200000:2.0:1789760742.349330:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.349342:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6049, 6049] 00020000:00200000:2.0:1789760742.349349:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24776704 chunk: [24776704, 24780800) 24780800, 4096 00020000:00200000:2.0:1789760742.349353:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.349368:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6049, 6049] 00000008:00010000:2.0:1789760742.349379:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.349384:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760742.349392:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24776704, 24780800) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.349402:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24776704=0x17a1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.349458:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.349461:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24780800 00000020:00010000:2.0:1789760742.349467:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.349470:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.349473:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.349476:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.349479:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.349485:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.349490:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.349502:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760742.349578:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 24776704<>24780800 00000100:00100000:3.0:1789760742.349631:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512634880:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760742.401976:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973368 gen 3 00000008:00010000:3.0:1789760742.402012:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760742.402018:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.402027:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.402065:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000080:00200000:2.0:1789760742.402079:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6049, 6049] 00000100:00100000:3.0:1789760742.402090:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512634880:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760742.402093:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.402097:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.402109:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24780800 i_blocks 49159 00000020:00200000:2.0:1789760742.402112:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.402119:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.402145:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24780800, count: 4096 00000080:00200000:2.0:1789760742.402151:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24780800, bytes: 4096 00000080:00200000:2.0:1789760742.402159:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.402166:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.402173:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24780800, 24784896) 0 0 00000080:00200000:2.0:1789760742.402177:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.402191:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6050, 6050] 00020000:00200000:2.0:1789760742.402197:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24780800 chunk: [24780800, 24784896) 24784896, 4096 00020000:00200000:2.0:1789760742.402201:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.402216:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6050, 6050] 00000008:00010000:2.0:1789760742.402229:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.402234:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760742.402244:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24780800, 24784896) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.402255:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24780800=0x17a2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.402298:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.402301:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24784896 00000020:00010000:2.0:1789760742.402307:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.402310:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.402313:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.402316:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.402319:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.402326:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.402331:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.402345:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760742.402430:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 24780800<>24784896 00000100:00100000:3.0:1789760742.402642:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760742.402654:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512635008:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760742.444583:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973369 gen 3 00000008:00010000:3.0:1789760742.444616:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760742.444622:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.444630:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.444647:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760742.444669:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512635008:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760742.445781:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6050, 6050] 00000080:00200000:2.0:1789760742.445796:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.445801:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.445813:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24784896 i_blocks 49159 00000020:00200000:2.0:1789760742.445816:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.445824:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.445846:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24784896, count: 4096 00000080:00200000:2.0:1789760742.445852:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24784896, bytes: 4096 00000080:00200000:2.0:1789760742.445861:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.445868:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.445870:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24784896, 24788992) 0 0 00000080:00200000:2.0:1789760742.445874:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.445885:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6051, 6051] 00020000:00200000:2.0:1789760742.445892:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24784896 chunk: [24784896, 24788992) 24788992, 4096 00020000:00200000:2.0:1789760742.445895:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.445934:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6051, 6051] 00000008:00010000:2.0:1789760742.445956:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.445961:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760742.445970:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24784896, 24788992) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.445980:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24784896=0x17a3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.446020:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.446023:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24788992 00000020:00010000:2.0:1789760742.446029:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.446032:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.446035:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.446038:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.446041:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.446047:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.446052:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.446067:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760742.446138:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 24784896<>24788992 00000100:00100000:3.0:1789760742.446406:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512635136:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760742.475939:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760742.475951:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.475960:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.475977:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760742.476001:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512635136:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760742.476218:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6051, 6051] 00000080:00200000:2.0:1789760742.476231:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760742.476236:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760742.476247:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24788992 i_blocks 49159 00000020:00200000:2.0:1789760742.476250:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760742.476258:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760742.476283:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24788992, count: 4096 00000080:00200000:2.0:1789760742.476289:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24788992, bytes: 4096 00000080:00200000:2.0:1789760742.476298:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760742.476305:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760742.476307:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24788992, 24793088) 0 0 00000080:00200000:2.0:1789760742.476311:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760742.476324:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6052, 6052] 00020000:00200000:2.0:1789760742.476330:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24788992 chunk: [24788992, 24793088) 24793088, 4096 00020000:00200000:2.0:1789760742.476333:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760742.476347:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6052, 6052] 00000008:00010000:2.0:1789760742.476359:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760742.476364:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760742.476373:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24788992, 24793088) append 0 append lockpos 0 00000080:00200000:2.0:1789760742.476384:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24788992=0x17a4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760742.476438:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760742.476441:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24793088 00000020:00010000:2.0:1789760742.476447:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760742.476450:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.476453:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760742.476456:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760742.476458:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760742.476475:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760742.476481:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760742.476493:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760742.476751:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 24788992<>24793088 00000100:00100000:2.0:1789760742.476780:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760742.476794:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512635264:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760742.507257:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973371 gen 3 00000008:00010000:3.0:1789760742.507310:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760742.507316:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.507324:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.507342:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760742.507368:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512635264:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760742.507817:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512635392:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760742.534177:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973372 gen 3 00000008:00010000:1.0:1789760742.534213:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760742.534220:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.534229:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.534246:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760742.534272:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512635392:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760742.535024:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760742.535039:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512635520:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760742.576077:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760742.576087:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.576098:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.576166:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760742.576188:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512635520:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760742.576840:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512635648:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760742.611935:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973373 gen 3 00000100:00100000:1.0:1789760742.611963:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005149180 x1876699512635648/t4294973374(4294973374) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760758 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760742.612002:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760742.612006:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.612012:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.612089:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760742.612111:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512635648:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760742.614261:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6055, 6055] 00000080:00200000:3.0:1789760742.614280:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760742.614286:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760742.614298:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24805376 i_blocks 49159 00000020:00200000:3.0:1789760742.614303:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760742.614314:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760742.614341:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24805376, count: 4096 00000080:00200000:3.0:1789760742.614349:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24805376, bytes: 4096 00000080:00200000:3.0:1789760742.614359:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760742.614366:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760742.614369:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24805376, 24809472) 0 0 00000080:00200000:3.0:1789760742.614372:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760742.614384:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6056, 6056] 00020000:00200000:3.0:1789760742.614391:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24805376 chunk: [24805376, 24809472) 24809472, 4096 00020000:00200000:3.0:1789760742.614395:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760742.614410:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6056, 6056] 00000008:00010000:3.0:1789760742.614437:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760742.614442:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760742.614450:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24805376, 24809472) append 0 append lockpos 0 00000080:00200000:3.0:1789760742.614461:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24805376=0x17a8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760742.614581:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760742.614584:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24809472 00000020:00010000:3.0:1789760742.614591:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760742.614594:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760742.614597:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760742.614600:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760742.614610:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760742.614618:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760742.614623:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760742.614647:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760742.614726:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 24805376<>24809472 00000100:00100000:3.0:1789760742.614773:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512635776:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760742.639960:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973374 gen 3 00000100:00100000:3.0:1789760742.639987:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc0107aed80 x1876699512635776/t4294973375(4294973375) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760758 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760742.640021:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:3.0:1789760742.640027:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.640035:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.640104:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:3.0:1789760742.640124:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512635776:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760742.643143:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6056, 6056] 00000080:00200000:1.0:1789760742.643163:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760742.643169:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760742.643185:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24809472 i_blocks 49159 00000020:00200000:1.0:1789760742.643190:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760742.643205:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760742.643230:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24809472, count: 4096 00000080:00200000:1.0:1789760742.643237:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24809472, bytes: 4096 00000080:00200000:1.0:1789760742.643249:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760742.643257:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760742.643259:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24809472, 24813568) 0 0 00000080:00200000:1.0:1789760742.643263:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760742.643277:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6057, 6057] 00020000:00200000:1.0:1789760742.643285:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24809472 chunk: [24809472, 24813568) 24813568, 4096 00020000:00200000:1.0:1789760742.643288:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760742.643305:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6057, 6057] 00000008:00010000:1.0:1789760742.643320:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760742.643325:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760742.643334:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24809472, 24813568) append 0 append lockpos 0 00000080:00200000:1.0:1789760742.643345:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24809472=0x17a9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760742.643399:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760742.643401:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24813568 00000020:00010000:1.0:1789760742.643408:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760742.643411:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.643425:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.643429:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760742.643431:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760742.643439:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760742.643445:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760742.643826:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760742.643954:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 24809472<>24813568 00000100:00100000:1.0:1789760742.643988:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760742.644007:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512635904:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760742.687449:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973376 gen 3 00000008:00010000:1.0:1789760742.687508:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760742.687515:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.687522:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.687538:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760742.687559:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512635904:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760742.687785:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6057, 6057] 00000080:00200000:3.0:1789760742.687799:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760742.687804:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760742.687891:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24813568 i_blocks 49159 00000020:00200000:3.0:1789760742.687896:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760742.687947:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760742.687973:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24813568, count: 4096 00000080:00200000:3.0:1789760742.687979:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24813568, bytes: 4096 00000080:00200000:3.0:1789760742.687986:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760742.687994:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760742.687996:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24813568, 24817664) 0 0 00000080:00200000:3.0:1789760742.688000:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760742.688012:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6058, 6058] 00020000:00200000:3.0:1789760742.688018:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24813568 chunk: [24813568, 24817664) 24817664, 4096 00020000:00200000:3.0:1789760742.688022:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760742.688037:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6058, 6058] 00000008:00010000:3.0:1789760742.688051:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760742.688055:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760742.688065:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24813568, 24817664) append 0 append lockpos 0 00000080:00200000:3.0:1789760742.688076:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24813568=0x17aa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760742.688126:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760742.688129:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24817664 00000020:00010000:3.0:1789760742.688135:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760742.688138:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760742.688142:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760742.688145:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760742.688147:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760742.688153:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760742.688158:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760742.688318:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760742.688394:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad880 - object 0x240000400:5 offset 24813568<>24817664 00000100:00100000:3.0:1789760742.688444:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760742.688458:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512636032:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760742.723698:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973377 gen 3 00000008:00010000:3.0:1789760742.723735:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:3.0:1789760742.723741:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.723751:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.723775:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:3.0:1789760742.723798:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512636032:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760742.723965:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6058, 6058] 00000080:00200000:1.0:1789760742.723979:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760742.723984:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760742.723997:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24817664 i_blocks 49159 00000020:00200000:1.0:1789760742.724001:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760742.724009:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760742.724037:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24817664, count: 4096 00000080:00200000:1.0:1789760742.724043:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24817664, bytes: 4096 00000080:00200000:1.0:1789760742.724051:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760742.724057:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760742.724059:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24817664, 24821760) 0 0 00000080:00200000:1.0:1789760742.724063:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760742.724077:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6059, 6059] 00020000:00200000:1.0:1789760742.724084:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24817664 chunk: [24817664, 24821760) 24821760, 4096 00020000:00200000:1.0:1789760742.724087:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760742.724102:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6059, 6059] 00000008:00010000:1.0:1789760742.724117:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760742.724122:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760742.724131:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24817664, 24821760) append 0 append lockpos 0 00000080:00200000:1.0:1789760742.724143:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24817664=0x17ab000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760742.724189:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760742.724192:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24821760 00000020:00010000:1.0:1789760742.724199:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760742.724202:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.724205:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.724208:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760742.724211:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760742.724217:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760742.724223:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760742.724237:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760742.724396:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 24817664<>24821760 00000080:00200000:1.0:1789760742.764776:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6059, 6059] 00000080:00200000:1.0:1789760742.764787:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760742.764793:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760742.764804:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24821760 i_blocks 49159 00000020:00200000:1.0:1789760742.764807:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760742.764815:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760742.764840:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24821760, count: 4096 00000080:00200000:1.0:1789760742.764847:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24821760, bytes: 4096 00000080:00200000:1.0:1789760742.764854:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760742.764862:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760742.764864:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24821760, 24825856) 0 0 00000080:00200000:1.0:1789760742.764868:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760742.764883:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6060, 6060] 00020000:00200000:1.0:1789760742.764890:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24821760 chunk: [24821760, 24825856) 24825856, 4096 00020000:00200000:1.0:1789760742.764894:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760742.764936:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6060, 6060] 00000008:00010000:1.0:1789760742.764950:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760742.764954:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760742.764964:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24821760, 24825856) append 0 append lockpos 0 00000080:00200000:1.0:1789760742.764974:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24821760=0x17ac000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760742.765018:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760742.765021:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24825856 00000020:00010000:1.0:1789760742.765028:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760742.765031:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.765034:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.765037:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760742.765039:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760742.765045:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760742.765051:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760742.765065:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760742.765141:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95c00 - object 0x240000400:5 offset 24821760<>24825856 00000080:00200000:1.0:1789760742.807440:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6060, 6060] 00000080:00200000:1.0:1789760742.807454:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760742.807459:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760742.807470:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24825856 i_blocks 49159 00000020:00200000:1.0:1789760742.807474:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760742.807481:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760742.807508:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24825856, count: 4096 00000080:00200000:1.0:1789760742.807514:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24825856, bytes: 4096 00000080:00200000:1.0:1789760742.807521:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760742.807528:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760742.807531:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24825856, 24829952) 0 0 00000080:00200000:1.0:1789760742.807547:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760742.807561:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6061, 6061] 00020000:00200000:1.0:1789760742.807568:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24825856 chunk: [24825856, 24829952) 24829952, 4096 00020000:00200000:1.0:1789760742.807572:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760742.807588:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6061, 6061] 00000008:00010000:1.0:1789760742.807601:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760742.807606:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760742.807616:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24825856, 24829952) append 0 append lockpos 0 00000080:00200000:1.0:1789760742.807627:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24825856=0x17ad000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760742.807671:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760742.807674:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24829952 00000020:00010000:1.0:1789760742.807681:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760742.807684:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.807687:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.807690:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760742.807693:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760742.807699:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760742.807705:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760742.807718:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760742.807795:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 24825856<>24829952 00000080:00200000:1.0:1789760742.850506:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6061, 6061] 00000080:00200000:1.0:1789760742.850520:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760742.850525:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760742.850538:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24829952 i_blocks 49159 00000020:00200000:1.0:1789760742.850541:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760742.850549:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760742.850576:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24829952, count: 4096 00000080:00200000:1.0:1789760742.850582:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24829952, bytes: 4096 00000080:00200000:1.0:1789760742.850592:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760742.850599:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760742.850601:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24829952, 24834048) 0 0 00000080:00200000:1.0:1789760742.850605:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760742.850620:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6062, 6062] 00020000:00200000:1.0:1789760742.850627:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24829952 chunk: [24829952, 24834048) 24834048, 4096 00020000:00200000:1.0:1789760742.850631:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760742.850648:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6062, 6062] 00000008:00010000:1.0:1789760742.850662:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760742.850668:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760742.850678:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24829952, 24834048) append 0 append lockpos 0 00000080:00200000:1.0:1789760742.852680:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24829952=0x17ae000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760742.853036:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760742.853039:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24834048 00000020:00010000:1.0:1789760742.853048:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760742.853051:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.853055:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.853058:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760742.853061:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760742.853180:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760742.853187:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760742.853213:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760742.853305:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 24829952<>24834048 00000100:00100000:1.0:1789760742.898890:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973381 gen 3 00000008:00010000:1.0:1789760742.898949:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760742.898955:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.898962:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.898979:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760742.898996:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512636544:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760742.899060:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6062, 6062] 00000080:00200000:3.0:1789760742.899077:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760742.899081:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760742.899091:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24834048 i_blocks 49159 00000020:00200000:3.0:1789760742.899094:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760742.899100:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760742.899122:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24834048, count: 4096 00000080:00200000:3.0:1789760742.899127:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24834048, bytes: 4096 00000080:00200000:3.0:1789760742.899133:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760742.899138:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760742.899139:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24834048, 24838144) 0 0 00000080:00200000:3.0:1789760742.899142:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760742.899153:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6063, 6063] 00020000:00200000:3.0:1789760742.899158:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24834048 chunk: [24834048, 24838144) 24838144, 4096 00020000:00200000:3.0:1789760742.899161:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760742.899172:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6063, 6063] 00000008:00010000:3.0:1789760742.899182:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760742.899187:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760742.899362:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24834048, 24838144) append 0 append lockpos 0 00000080:00200000:3.0:1789760742.899373:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24834048=0x17af000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760742.899407:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760742.899423:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24838144 00000020:00010000:3.0:1789760742.899431:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760742.899434:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760742.899437:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760742.899440:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760742.899443:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760742.899448:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760742.899453:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760742.899468:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760742.899542:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad880 - object 0x240000400:5 offset 24834048<>24838144 00000100:00100000:2.0:1789760742.899597:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760742.899661:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512636672:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760742.931397:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973382 gen 3 00000008:00010000:3.0:1789760742.931450:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:3.0:1789760742.931456:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760742.931465:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760742.931488:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:3.0:1789760742.931510:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512636672:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760742.932127:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6063, 6063] 00000080:00200000:1.0:1789760742.932141:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760742.932146:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760742.932159:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24838144 i_blocks 49159 00000020:00200000:1.0:1789760742.932162:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760742.932177:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760742.932203:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24838144, count: 4096 00000080:00200000:1.0:1789760742.932210:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24838144, bytes: 4096 00000080:00200000:1.0:1789760742.932220:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760742.932227:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760742.932229:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24838144, 24842240) 0 0 00000080:00200000:1.0:1789760742.932233:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760742.932246:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6064, 6064] 00020000:00200000:1.0:1789760742.932252:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24838144 chunk: [24838144, 24842240) 24842240, 4096 00020000:00200000:1.0:1789760742.932256:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760742.932269:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6064, 6064] 00000008:00010000:1.0:1789760742.932282:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760742.932287:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760742.932295:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24838144, 24842240) append 0 append lockpos 0 00000080:00200000:1.0:1789760742.932305:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24838144=0x17b0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760742.932348:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760742.932351:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24842240 00000020:00010000:1.0:1789760742.932357:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760742.932360:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.932362:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760742.932365:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760742.932380:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760742.932387:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760742.932393:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760742.932407:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760742.932502:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 24838144<>24842240 00000100:00100000:1.0:1789760742.932544:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760742.932559:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512636800:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760742.973960:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973383 gen 3 00000008:00010000:1.0:1789760742.973998:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760742.974004:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760742.974012:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760742.974035:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760742.974060:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512636800:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760742.975432:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6064, 6064] 00000080:00200000:3.0:1789760742.975448:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760742.975453:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760742.975467:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24842240 i_blocks 49159 00000020:00200000:3.0:1789760742.975471:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760742.975482:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760742.975508:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24842240, count: 4096 00000080:00200000:3.0:1789760742.975515:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24842240, bytes: 4096 00000080:00200000:3.0:1789760742.975525:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760742.975533:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760742.975535:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24842240, 24846336) 0 0 00000080:00200000:3.0:1789760742.975539:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760742.975552:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6065, 6065] 00020000:00200000:3.0:1789760742.975559:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24842240 chunk: [24842240, 24846336) 24846336, 4096 00020000:00200000:3.0:1789760742.975563:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760742.975579:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6065, 6065] 00000008:00010000:3.0:1789760742.975593:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760742.975598:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760742.975607:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24842240, 24846336) append 0 append lockpos 0 00000080:00200000:3.0:1789760742.975618:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24842240=0x17b1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760742.975668:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760742.975671:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24846336 00000020:00010000:3.0:1789760742.975678:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760742.975681:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760742.975685:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760742.975688:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760742.975691:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760742.975698:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760742.975704:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760742.975724:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760742.975815:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad880 - object 0x240000400:5 offset 24842240<>24846336 00000100:00100000:3.0:1789760742.975869:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512636928:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760743.009714:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:3.0:1789760743.009726:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760743.009736:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760743.009758:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:3.0:1789760743.009782:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512636928:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760743.009926:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6065, 6065] 00000080:00200000:1.0:1789760743.009939:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760743.009944:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760743.009955:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24846336 i_blocks 49159 00000020:00200000:1.0:1789760743.009958:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760743.009966:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760743.009991:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24846336, count: 4096 00000080:00200000:1.0:1789760743.009997:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24846336, bytes: 4096 00000080:00200000:1.0:1789760743.010006:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760743.010013:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760743.010015:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24846336, 24850432) 0 0 00000080:00200000:1.0:1789760743.010019:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760743.010031:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6066, 6066] 00020000:00200000:1.0:1789760743.010038:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24846336 chunk: [24846336, 24850432) 24850432, 4096 00020000:00200000:1.0:1789760743.010042:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760743.010127:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6066, 6066] 00000008:00010000:1.0:1789760743.010142:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760743.010146:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760743.010157:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24846336, 24850432) append 0 append lockpos 0 00000080:00200000:1.0:1789760743.010170:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24846336=0x17b2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760743.010214:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760743.010217:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24850432 00000020:00010000:1.0:1789760743.010224:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760743.010227:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.010230:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.010233:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760743.010235:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760743.010242:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760743.010247:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760743.010263:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760743.010338:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 24846336<>24850432 00000080:00200000:1.0:1789760743.052332:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6066, 6066] 00000080:00200000:1.0:1789760743.052348:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760743.052353:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760743.052367:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24850432 i_blocks 49159 00000020:00200000:1.0:1789760743.052371:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760743.052379:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760743.052429:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24850432, count: 4096 00000080:00200000:1.0:1789760743.052437:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24850432, bytes: 4096 00000080:00200000:1.0:1789760743.052447:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760743.052455:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760743.052457:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24850432, 24854528) 0 0 00000080:00200000:1.0:1789760743.052461:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760743.052475:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6067, 6067] 00020000:00200000:1.0:1789760743.052483:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24850432 chunk: [24850432, 24854528) 24854528, 4096 00020000:00200000:1.0:1789760743.052488:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760743.052505:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6067, 6067] 00000008:00010000:1.0:1789760743.052521:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760743.052526:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760743.052536:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24850432, 24854528) append 0 append lockpos 0 00000080:00200000:1.0:1789760743.052548:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24850432=0x17b3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760743.052597:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760743.052600:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24854528 00000020:00010000:1.0:1789760743.052607:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760743.052610:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.052613:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.052616:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760743.052620:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760743.052629:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760743.052638:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760743.052658:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760743.052762:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97100 - object 0x240000400:5 offset 24850432<>24854528 00000080:00200000:1.0:1789760743.085254:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6067, 6067] 00000080:00200000:1.0:1789760743.085265:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760743.085270:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760743.085280:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24854528 i_blocks 49159 00000020:00200000:1.0:1789760743.085283:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760743.085290:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760743.085314:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24854528, count: 4096 00000080:00200000:1.0:1789760743.085320:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24854528, bytes: 4096 00000080:00200000:1.0:1789760743.085328:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760743.085335:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760743.085337:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24854528, 24858624) 0 0 00000080:00200000:1.0:1789760743.085341:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760743.085353:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6068, 6068] 00020000:00200000:1.0:1789760743.085360:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24854528 chunk: [24854528, 24858624) 24858624, 4096 00020000:00200000:1.0:1789760743.085363:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760743.085378:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6068, 6068] 00000008:00010000:1.0:1789760743.085400:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760743.085405:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760743.085423:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24854528, 24858624) append 0 append lockpos 0 00000080:00200000:1.0:1789760743.085433:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24854528=0x17b4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760743.085476:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760743.085479:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24858624 00000020:00010000:1.0:1789760743.085486:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760743.085489:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.085492:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.085495:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760743.085498:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760743.085504:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760743.085511:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760743.085524:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760743.085596:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 24854528<>24858624 00000080:00200000:1.0:1789760743.116878:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6068, 6068] 00000080:00200000:1.0:1789760743.116892:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760743.116898:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760743.116938:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24858624 i_blocks 49159 00000020:00200000:1.0:1789760743.116942:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760743.116950:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760743.116976:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24858624, count: 4096 00000080:00200000:1.0:1789760743.116983:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24858624, bytes: 4096 00000080:00200000:1.0:1789760743.116991:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760743.116997:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760743.116999:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24858624, 24862720) 0 0 00000080:00200000:1.0:1789760743.117003:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760743.117015:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6069, 6069] 00020000:00200000:1.0:1789760743.117022:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24858624 chunk: [24858624, 24862720) 24862720, 4096 00020000:00200000:1.0:1789760743.117025:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760743.117044:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6069, 6069] 00000008:00010000:1.0:1789760743.117070:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760743.117075:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760743.117084:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24858624, 24862720) append 0 append lockpos 0 00000080:00200000:1.0:1789760743.117095:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24858624=0x17b5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760743.117140:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760743.117143:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24862720 00000020:00010000:1.0:1789760743.117150:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760743.117153:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.117156:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.117159:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760743.117162:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760743.117168:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760743.117173:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760743.117186:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760743.117438:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 24858624<>24862720 00000080:00200000:1.0:1789760743.150668:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6069, 6069] 00000080:00200000:1.0:1789760743.150681:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760743.150686:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760743.150698:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24862720 i_blocks 49159 00000020:00200000:1.0:1789760743.150700:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760743.150709:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760743.150734:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24862720, count: 4096 00000080:00200000:1.0:1789760743.150740:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24862720, bytes: 4096 00000080:00200000:1.0:1789760743.150748:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760743.150754:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760743.150756:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24862720, 24866816) 0 0 00000080:00200000:1.0:1789760743.150760:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760743.150773:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6070, 6070] 00020000:00200000:1.0:1789760743.150780:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24862720 chunk: [24862720, 24866816) 24866816, 4096 00020000:00200000:1.0:1789760743.150785:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760743.150864:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6070, 6070] 00000008:00010000:1.0:1789760743.150879:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760743.150884:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760743.150893:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24862720, 24866816) append 0 append lockpos 0 00000080:00200000:1.0:1789760743.150925:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24862720=0x17b6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760743.150971:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760743.150974:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24866816 00000020:00010000:1.0:1789760743.150981:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760743.150984:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.150987:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.150990:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760743.150992:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760743.150998:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760743.151003:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760743.151017:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760743.151096:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97100 - object 0x240000400:5 offset 24862720<>24866816 00000080:00200000:1.0:1789760743.196990:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6070, 6070] 00000080:00200000:1.0:1789760743.197003:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760743.197009:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760743.197020:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24866816 i_blocks 49159 00000020:00200000:1.0:1789760743.197024:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760743.197037:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00010000:00010000:2.0:1789760743.197058:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1789760743.197063:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24866816, count: 4096 00000080:00200000:1.0:1789760743.197071:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24866816, bytes: 4096 00000080:00200000:1.0:1789760743.197080:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760743.197087:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760743.197089:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24866816, 24870912) 0 0 00000080:00200000:1.0:1789760743.197093:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760743.197107:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6071, 6071] 00020000:00200000:1.0:1789760743.197114:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24866816 chunk: [24866816, 24870912) 24870912, 4096 00020000:00200000:1.0:1789760743.197118:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760743.197133:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6071, 6071] 00000008:00010000:1.0:1789760743.197146:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760743.197152:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760743.197174:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24866816, 24870912) append 0 append lockpos 0 00000080:00200000:1.0:1789760743.197185:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24866816=0x17b7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760743.197232:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760743.197236:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24870912 00000020:00010000:1.0:1789760743.197243:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760743.197246:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.197249:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.197252:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760743.197255:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760743.197261:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760743.197267:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760743.197281:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760743.197357:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 24866816<>24870912 00000100:00100000:1.0:1789760743.197393:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760743.197408:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512637696:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.234302:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973390 gen 3 00000008:00010000:1.0:1789760743.234338:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760743.234344:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.234352:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.234377:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760743.234400:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512637696:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.235483:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512637824:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.289330:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973391 gen 3 00000008:00010000:1.0:1789760743.289368:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760743.289375:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.289384:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.289408:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760743.289442:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512637824:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.291062:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512637952:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.328128:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973392 gen 3 00000008:00010000:1.0:1789760743.328169:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760743.328177:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.328187:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.328210:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760743.328235:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512637952:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.328753:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760743.328765:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512638080:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.366219:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973393 gen 3 00000008:00010000:1.0:1789760743.366254:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760743.366260:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.366269:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.366387:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760743.366421:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512638080:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.367226:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512638208:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.404304:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973394 gen 3 00000008:00010000:1.0:1789760743.404342:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760743.404348:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.404356:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.404372:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760743.404393:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512638208:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760743.404428:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6075, 6075] 00000080:00200000:3.0:1789760743.404441:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760743.404445:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760743.404457:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24887296 i_blocks 49159 00000020:00200000:3.0:1789760743.404460:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760743.404468:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760743.404498:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24887296, count: 4096 00000080:00200000:3.0:1789760743.404504:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24887296, bytes: 4096 00000080:00200000:3.0:1789760743.404511:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760743.404518:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760743.404520:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24887296, 24891392) 0 0 00000080:00200000:3.0:1789760743.404523:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760743.404536:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6076, 6076] 00020000:00200000:3.0:1789760743.404542:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24887296 chunk: [24887296, 24891392) 24891392, 4096 00020000:00200000:3.0:1789760743.404545:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760743.404559:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6076, 6076] 00000008:00010000:3.0:1789760743.404572:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760743.404577:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760743.404585:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24887296, 24891392) append 0 append lockpos 0 00000080:00200000:3.0:1789760743.404596:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24887296=0x17bc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760743.404635:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760743.404637:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24891392 00000020:00010000:3.0:1789760743.404642:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760743.404644:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760743.404646:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760743.404648:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760743.404650:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760743.404654:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760743.404658:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760743.404669:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760743.404739:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adf80 - object 0x240000400:5 offset 24887296<>24891392 00000100:00100000:3.0:1789760743.404775:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760743.404788:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512638336:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760743.435441:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973395 gen 3 00000008:00010000:3.0:1789760743.435477:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:3.0:1789760743.435483:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760743.435491:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760743.435631:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:3.0:1789760743.435657:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512638336:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.438961:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512638464:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.471599:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973396 gen 3 00000008:00010000:1.0:1789760743.471638:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760743.471645:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.471654:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.471825:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760743.471850:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512638464:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.472388:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512638592:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.506225:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973397 gen 3 00000008:00010000:1.0:1789760743.506259:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760743.506265:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.506275:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.506297:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760743.506326:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512638592:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.506823:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760743.506837:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512638720:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.560651:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973398 gen 3 00000008:00010000:1.0:1789760743.560697:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760743.560704:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.560713:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.560733:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760743.560757:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512638720:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.562040:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512638848:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.633609:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973399 gen 3 00000008:00010000:1.0:1789760743.633646:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760743.633654:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.633663:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.633684:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760743.633708:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512638848:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.639719:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512638976:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.672131:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973400 gen 3 00000008:00010000:1.0:1789760743.672166:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760743.672172:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.672180:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.672252:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760743.672286:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512638976:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.673503:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512639104:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760743.716378:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760743.716391:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.716401:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.716433:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760743.716456:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512639104:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.718426:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760743.718448:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512639232:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.767200:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973402 gen 3 00000008:00010000:1.0:1789760743.767246:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760743.767253:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.767261:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.767282:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760743.767305:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512639232:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.767936:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512639360:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.814040:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973403 gen 3 00000008:00010000:1.0:1789760743.814076:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760743.814083:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.814093:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.814116:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760743.814139:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512639360:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.815757:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760743.815777:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512639488:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.860229:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973404 gen 3 00000008:00010000:1.0:1789760743.860268:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760743.860275:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.860285:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.860311:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760743.860335:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512639488:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760743.860794:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512639616:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760743.900961:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760743.900972:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760743.900981:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760743.901003:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760743.901025:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512639616:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760743.952799:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6087, 6087] 00000080:00200000:1.0:1789760743.952811:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760743.952816:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760743.952826:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24936448 i_blocks 49159 00000020:00200000:1.0:1789760743.952829:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760743.952954:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760743.952984:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24936448, count: 4096 00000080:00200000:1.0:1789760743.952990:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24936448, bytes: 4096 00000080:00200000:1.0:1789760743.952998:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760743.953007:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760743.953009:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24936448, 24940544) 0 0 00000080:00200000:1.0:1789760743.953013:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760743.953026:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6088, 6088] 00020000:00200000:1.0:1789760743.953034:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24936448 chunk: [24936448, 24940544) 24940544, 4096 00020000:00200000:1.0:1789760743.953038:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760743.953052:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6088, 6088] 00000008:00010000:1.0:1789760743.953065:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760743.953070:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760743.953079:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24936448, 24940544) append 0 append lockpos 0 00000080:00200000:1.0:1789760743.953089:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24936448=0x17c8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760743.953131:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760743.953134:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24940544 00000020:00010000:1.0:1789760743.953140:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760743.953143:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.953145:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760743.953147:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760743.953150:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760743.953155:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760743.953161:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760743.953175:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760743.953244:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 24936448<>24940544 00000080:00200000:1.0:1789760744.006943:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6088, 6088] 00000080:00200000:1.0:1789760744.006960:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760744.006966:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760744.006980:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24940544 i_blocks 49159 00000020:00200000:1.0:1789760744.006985:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760744.006995:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760744.007020:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24940544, count: 4096 00000080:00200000:1.0:1789760744.007026:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24940544, bytes: 4096 00000080:00200000:1.0:1789760744.007037:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760744.007045:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760744.007047:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24940544, 24944640) 0 0 00000080:00200000:1.0:1789760744.007051:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760744.007064:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6089, 6089] 00020000:00200000:1.0:1789760744.007072:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24940544 chunk: [24940544, 24944640) 24944640, 4096 00020000:00200000:1.0:1789760744.007075:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760744.007091:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6089, 6089] 00000008:00010000:1.0:1789760744.007105:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760744.007109:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760744.007118:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24940544, 24944640) append 0 append lockpos 0 00000080:00200000:1.0:1789760744.007131:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24940544=0x17c9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760744.007183:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760744.007186:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24944640 00000020:00010000:1.0:1789760744.007192:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760744.007195:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760744.007197:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760744.007200:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760744.007202:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760744.007208:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760744.007213:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760744.007234:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760744.007312:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 24940544<>24944640 00000080:00200000:1.0:1789760744.050069:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6089, 6089] 00000080:00200000:1.0:1789760744.050081:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760744.050086:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760744.050097:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24944640 i_blocks 49159 00000020:00200000:1.0:1789760744.050100:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760744.050108:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760744.050132:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24944640, count: 4096 00000080:00200000:1.0:1789760744.050138:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24944640, bytes: 4096 00000080:00200000:1.0:1789760744.050146:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760744.050154:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760744.050156:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24944640, 24948736) 0 0 00000080:00200000:1.0:1789760744.050160:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760744.050174:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6090, 6090] 00020000:00200000:1.0:1789760744.050180:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24944640 chunk: [24944640, 24948736) 24948736, 4096 00020000:00200000:1.0:1789760744.050183:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760744.050198:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6090, 6090] 00000008:00010000:1.0:1789760744.050209:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760744.050214:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760744.050222:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24944640, 24948736) append 0 append lockpos 0 00000080:00200000:1.0:1789760744.050234:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24944640=0x17ca000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760744.050276:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760744.050279:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24948736 00000020:00010000:1.0:1789760744.050285:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760744.050288:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760744.050291:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760744.050293:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760744.050296:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760744.050302:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760744.050307:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760744.050320:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760744.050400:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95c00 - object 0x240000400:5 offset 24944640<>24948736 00000080:00200000:3.0:1789760744.092556:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6090, 6090] 00000080:00200000:3.0:1789760744.092639:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760744.092646:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760744.092660:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24948736 i_blocks 49159 00000020:00200000:3.0:1789760744.092665:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760744.092679:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760744.092706:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24948736, count: 4096 00000080:00200000:3.0:1789760744.092713:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24948736, bytes: 4096 00000080:00200000:3.0:1789760744.092725:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760744.092733:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760744.092735:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24948736, 24952832) 0 0 00000080:00200000:3.0:1789760744.092739:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760744.092752:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6091, 6091] 00020000:00200000:3.0:1789760744.092759:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24948736 chunk: [24948736, 24952832) 24952832, 4096 00020000:00200000:3.0:1789760744.092762:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760744.092778:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6091, 6091] 00000008:00010000:3.0:1789760744.092953:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760744.092958:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760744.092969:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24948736, 24952832) append 0 append lockpos 0 00000080:00200000:3.0:1789760744.092980:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24948736=0x17cb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760744.093036:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760744.093039:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24952832 00000020:00010000:3.0:1789760744.093046:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760744.093049:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.093052:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.093055:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760744.093058:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760744.093064:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760744.093070:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760744.093672:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760744.093755:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adf80 - object 0x240000400:5 offset 24948736<>24952832 00000100:00100000:3.0:1789760744.093874:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512640256:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760744.142548:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973410 gen 3 00000008:00010000:3.0:1789760744.142593:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:3.0:1789760744.142600:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760744.142609:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760744.142636:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:3.0:1789760744.142660:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512640256:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760744.182842:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6092, 6092] 00000080:00200000:3.0:1789760744.182864:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760744.182872:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760744.182889:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24956928 i_blocks 49159 00000020:00200000:3.0:1789760744.182895:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760744.182926:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760744.182963:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24956928, count: 4096 00000080:00200000:3.0:1789760744.182971:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24956928, bytes: 4096 00000080:00200000:3.0:1789760744.182986:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760744.182995:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760744.182998:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24956928, 24961024) 0 0 00000080:00200000:3.0:1789760744.183002:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760744.183021:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6093, 6093] 00020000:00200000:3.0:1789760744.183031:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24956928 chunk: [24956928, 24961024) 24961024, 4096 00020000:00200000:3.0:1789760744.183035:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760744.183057:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6093, 6093] 00000008:00010000:3.0:1789760744.183093:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760744.183100:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760744.183113:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24956928, 24961024) append 0 append lockpos 0 00000080:00200000:3.0:1789760744.183523:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24956928=0x17cd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760744.183593:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760744.183597:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24961024 00000020:00010000:3.0:1789760744.183605:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760744.183609:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.183612:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.183616:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760744.183619:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760744.183626:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760744.183632:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760744.183651:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760744.183758:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adf80 - object 0x240000400:5 offset 24956928<>24961024 00000100:00100000:2.0:1789760744.183830:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512640512:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760744.221116:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e6cf7267-9b19-462a-a263-14115df15dab:2361:1876699512640640:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760744.221248:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2361:1876699512640896:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760744.221288:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760744.221295:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512640768:192.168.203.156@tcp:400:kworker.0 00000008:00010000:3.0:1789760744.222092:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:3.0:1789760744.222102:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760744.222114:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760744.222141:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:3.0:1789760744.222164:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512640512:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760744.222879:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760744.224139:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512641024:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760744.224668:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e6cf7267-9b19-462a-a263-14115df15dab:2361:1876699512640640:192.168.203.156@tcp:400:kworker.0 00000100:00100000:2.0:1789760744.231885:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512640768:192.168.203.156@tcp:400:kworker.0 00000100:00100000:2.0:1789760744.231941:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2361:1876699512640896:192.168.203.156@tcp:400:kworker.0 00000100:00100000:1.0:1789760744.276716:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973413 gen 3 00000008:00010000:1.0:1789760744.276803:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760744.276812:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.276824:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.276846:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760744.276875:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512641024:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760744.278399:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512641152:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760744.312747:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973414 gen 3 00000008:00010000:1.0:1789760744.312854:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760744.312864:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.312876:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.312982:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760744.313013:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512641152:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760744.313671:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512641280:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760744.351987:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973415 gen 3 00000008:00010000:1.0:1789760744.352047:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760744.352055:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.352368:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.352395:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000080:00200000:3.0:1789760744.352477:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6096, 6096] 00000080:00200000:3.0:1789760744.352495:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760744.352501:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760744.352517:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24973312 i_blocks 49159 00000020:00200000:3.0:1789760744.352520:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760744.352530:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000100:00100000:1.0:1789760744.352559:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512641280:192.168.203.156@tcp:4:dd.0 00000080:00200008:3.0:1789760744.352566:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24973312, count: 4096 00000080:00200000:3.0:1789760744.352574:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24973312, bytes: 4096 00000080:00200000:3.0:1789760744.352585:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760744.352593:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760744.352596:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24973312, 24977408) 0 0 00000080:00200000:3.0:1789760744.352599:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760744.352617:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6097, 6097] 00020000:00200000:3.0:1789760744.352625:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24973312 chunk: [24973312, 24977408) 24977408, 4096 00020000:00200000:3.0:1789760744.352631:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760744.352651:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6097, 6097] 00000008:00010000:3.0:1789760744.352669:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760744.352675:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760744.352688:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24973312, 24977408) append 0 append lockpos 0 00000080:00200000:3.0:1789760744.352703:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24973312=0x17d1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760744.352768:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760744.352771:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24977408 00000020:00010000:3.0:1789760744.352780:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760744.352784:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.352787:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.352993:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760744.352998:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760744.353007:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760744.353014:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760744.353030:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760744.353392:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad180 - object 0x240000400:5 offset 24973312<>24977408 00000100:00100000:2.0:1789760744.353512:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760744.353535:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512641408:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760744.397265:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973416 gen 3 00000008:00010000:3.0:1789760744.397324:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:3.0:1789760744.397333:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760744.397345:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760744.397361:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:3.0:1789760744.397392:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512641408:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760744.397709:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6097, 6097] 00000080:00200000:2.0:1789760744.397726:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760744.397732:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760744.397747:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24977408 i_blocks 49159 00000020:00200000:2.0:1789760744.397982:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760744.397993:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760744.398026:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24977408, count: 4096 00000080:00200000:2.0:1789760744.398034:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24977408, bytes: 4096 00000080:00200000:2.0:1789760744.398044:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760744.398052:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760744.398054:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24977408, 24981504) 0 0 00000080:00200000:2.0:1789760744.398058:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760744.398086:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6098, 6098] 00020000:00200000:2.0:1789760744.398095:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24977408 chunk: [24977408, 24981504) 24981504, 4096 00020000:00200000:2.0:1789760744.398099:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760744.398121:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6098, 6098] 00000008:00010000:2.0:1789760744.398139:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760744.398144:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760744.398155:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24977408, 24981504) append 0 append lockpos 0 00000080:00200000:2.0:1789760744.398168:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24977408=0x17d2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760744.398243:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760744.398246:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24981504 00000020:00010000:2.0:1789760744.398254:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760744.398258:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.398261:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.398264:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760744.398268:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760744.398275:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760744.398282:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760744.398306:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760744.398540:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 24977408<>24981504 00000100:00100000:2.0:1789760744.398586:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760744.398606:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512641536:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760744.441814:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973417 gen 3 00000008:00010000:3.0:1789760744.441874:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760744.441882:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760744.441894:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760744.442117:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760744.442212:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512641536:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760744.442467:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6098, 6098] 00000080:00200000:2.0:1789760744.442485:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760744.442491:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760744.442507:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 24981504 i_blocks 49159 00000020:00200000:2.0:1789760744.442512:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760744.442524:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760744.442562:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 24981504, count: 4096 00000080:00200000:2.0:1789760744.442570:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 24981504, bytes: 4096 00000080:00200000:2.0:1789760744.442581:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760744.442590:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760744.442592:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [24981504, 24985600) 0 0 00000080:00200000:2.0:1789760744.442596:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760744.442615:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6099, 6099] 00020000:00200000:2.0:1789760744.442626:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 24981504 chunk: [24981504, 24985600) 24985600, 4096 00020000:00200000:2.0:1789760744.442630:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760744.442652:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6099, 6099] 00000008:00010000:2.0:1789760744.442670:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760744.442676:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760744.442691:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [24981504, 24985600) append 0 append lockpos 0 00000080:00200000:2.0:1789760744.442706:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=24981504=0x17d3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760744.442769:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760744.442772:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 24985600 00000020:00010000:2.0:1789760744.443321:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760744.443326:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.443330:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760744.443333:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760744.443336:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760744.443345:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760744.443353:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760744.443377:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760744.443577:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 24981504<>24985600 00000100:00100000:2.0:1789760744.444330:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512641664:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760744.498706:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760744.498721:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760744.498732:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760744.498758:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760744.499716:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512641664:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760744.499751:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc300 00000008:00100000:3.0:1789760744.499987:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad180 - object 0x240000400:5 offset 24985600<>24989696 00000100:00100000:3.0:1789760744.500029:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760744.500043:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512641792:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760744.536414:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973419 gen 3 00000008:00010000:3.0:1789760744.536479:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:3.0:1789760744.536497:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760744.536508:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760744.536525:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:3.0:1789760744.536555:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512641792:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760744.537597:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512641920:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760744.581482:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973420 gen 3 00000008:00010000:1.0:1789760744.581535:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760744.581544:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.581557:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.581578:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760744.581608:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512641920:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760744.585942:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512642048:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760744.644269:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973421 gen 3 00000008:00010000:1.0:1789760744.644320:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760744.644330:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.644343:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.644370:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760744.644411:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512642048:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760744.646498:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512642176:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760744.700024:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973422 gen 3 00000008:00010000:1.0:1789760744.700076:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760744.700085:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.700097:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.700123:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760744.700154:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512642176:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760744.701304:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512642304:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760744.753778:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973423 gen 3 00000008:00010000:1.0:1789760744.753844:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760744.753852:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.753864:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.753888:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760744.753948:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512642304:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760744.758304:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512642432:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760744.830958:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973424 gen 3 00000008:00010000:1.0:1789760744.831010:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760744.831019:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760744.831032:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760744.831048:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760744.831078:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512642432:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760744.898523:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6106, 6106] 00000080:00200000:3.0:1789760744.898540:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760744.898547:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760744.898562:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25014272 i_blocks 49159 00000020:00200000:3.0:1789760744.898565:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760744.898575:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760744.898610:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25014272, count: 4096 00000080:00200000:3.0:1789760744.898617:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25014272, bytes: 4096 00000080:00200000:3.0:1789760744.898629:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760744.898637:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760744.898640:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25014272, 25018368) 0 0 00000080:00200000:3.0:1789760744.898645:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760744.898662:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6107, 6107] 00020000:00200000:3.0:1789760744.898671:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25014272 chunk: [25014272, 25018368) 25018368, 4096 00020000:00200000:3.0:1789760744.898675:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760744.898692:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6107, 6107] 00000008:00010000:3.0:1789760744.898710:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760744.898716:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760744.898727:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25014272, 25018368) append 0 append lockpos 0 00000080:00200000:3.0:1789760744.898741:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25014272=0x17db000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760744.899007:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760744.899011:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25018368 00000020:00010000:3.0:1789760744.899019:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760744.899022:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.899025:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760744.899029:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760744.899031:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760744.899039:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760744.899045:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760744.899065:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760744.899300:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad180 - object 0x240000400:5 offset 25014272<>25018368 00000100:00100000:3.0:1789760744.899353:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760744.899374:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512642688:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760744.971650:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973426 gen 3 00000008:00010000:3.0:1789760744.971707:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:3.0:1789760744.971717:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760744.971732:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760744.971824:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:3.0:1789760744.971857:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512642688:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760744.972719:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512642816:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.034975:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973427 gen 3 00000008:00010000:1.0:1789760745.035028:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760745.035037:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.035050:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.035074:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760745.035106:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512642816:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.036971:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512642944:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.095230:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973428 gen 3 00000008:00010000:1.0:1789760745.095283:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760745.095292:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.095304:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.095333:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760745.095363:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512642944:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760745.096167:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccb40 00000008:00100000:1.0:1789760745.096307:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 25026560<>25030656 00000100:00100000:1.0:1789760745.096356:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512643072:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.175388:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973429 gen 3 00000008:00010000:1.0:1789760745.175456:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760745.175465:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.175477:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.175500:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760745.175532:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512643072:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.176279:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512643200:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.243386:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973430 gen 3 00000008:00010000:1.0:1789760745.243451:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760745.243460:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.243474:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.243497:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760745.243527:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512643200:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.244301:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512643328:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.280344:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973431 gen 3 00000008:00010000:1.0:1789760745.280406:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760745.280415:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.280427:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.280449:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760745.280476:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512643328:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.281755:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512643456:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.384336:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973432 gen 3 00000008:00010000:1.0:1789760745.384392:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760745.384411:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.384424:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.384452:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760745.384481:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512643456:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.385737:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512643584:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760745.433572:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760745.433652:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.433666:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.433743:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760745.433776:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512643584:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.434582:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512643712:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.502596:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973434 gen 3 00000008:00010000:1.0:1789760745.502662:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760745.502670:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.502682:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.502711:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760745.502741:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512643712:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.507260:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512643840:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.585014:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973435 gen 3 00000008:00010000:1.0:1789760745.585192:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760745.585200:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.585212:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.585321:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760745.585352:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512643840:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.587178:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760745.587214:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512643968:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760745.657226:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760745.657241:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.657254:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.657324:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760745.657358:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512643968:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.658758:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760745.658797:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512644096:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.749346:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973437 gen 3 00000008:00010000:1.0:1789760745.749423:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760745.749432:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.749444:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.749467:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760745.749496:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512644096:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.750576:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760745.750597:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512644224:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.808800:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973438 gen 3 00000008:00010000:1.0:1789760745.808855:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760745.808864:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.808878:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.808941:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760745.808969:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512644224:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.810301:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512644352:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.876071:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973439 gen 3 00000008:00010000:1.0:1789760745.876125:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760745.876133:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.876602:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.876633:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760745.876669:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512644352:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.877608:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760745.877627:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512644480:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760745.932004:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760745.932021:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760745.932035:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760745.932053:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760745.932088:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512644480:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760745.933000:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512644608:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.007657:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973441 gen 3 00000008:00010000:1.0:1789760746.007725:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760746.007734:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.007745:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.007767:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760746.007803:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512644608:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760746.008372:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc180 00000008:00100000:1.0:1789760746.008488:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 25079808<>25083904 00000080:00200000:3.0:1789760746.050513:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6123, 6123] 00000080:00200000:3.0:1789760746.050530:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760746.050537:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760746.050553:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25083904 i_blocks 49159 00000020:00200000:3.0:1789760746.050556:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760746.050566:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760746.050601:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25083904, count: 4096 00000080:00200000:3.0:1789760746.050609:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25083904, bytes: 4096 00000080:00200000:3.0:1789760746.050621:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760746.050630:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760746.050632:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25083904, 25088000) 0 0 00000080:00200000:3.0:1789760746.050637:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760746.050653:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6124, 6124] 00020000:00200000:3.0:1789760746.050662:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25083904 chunk: [25083904, 25088000) 25088000, 4096 00020000:00200000:3.0:1789760746.050667:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760746.050687:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6124, 6124] 00000008:00010000:3.0:1789760746.050705:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760746.050711:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760746.050723:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25083904, 25088000) append 0 append lockpos 0 00000080:00200000:3.0:1789760746.050737:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25083904=0x17ec000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760746.050807:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760746.050810:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25088000 00000020:00010000:3.0:1789760746.050818:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760746.050822:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760746.050826:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760746.050976:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760746.050979:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760746.050988:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760746.050995:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760746.051014:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760746.051131:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad180 - object 0x240000400:5 offset 25083904<>25088000 00000100:00100000:2.0:1789760746.051300:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512644864:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760746.093657:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973443 gen 3 00000008:00010000:3.0:1789760746.093704:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:3.0:1789760746.093713:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.093725:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.093749:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:3.0:1789760746.093777:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512644864:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.095199:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760746.095581:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512644992:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.136187:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973444 gen 3 00000008:00010000:1.0:1789760746.136240:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760746.136249:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.136262:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.136339:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760746.136372:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512644992:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760746.136470:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6125, 6125] 00000080:00200000:3.0:1789760746.136487:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760746.136493:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760746.136509:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25092096 i_blocks 49159 00000020:00200000:3.0:1789760746.136513:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760746.136523:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760746.136557:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25092096, count: 4096 00000080:00200000:3.0:1789760746.136565:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25092096, bytes: 4096 00000080:00200000:3.0:1789760746.136576:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760746.136585:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760746.136587:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25092096, 25096192) 0 0 00000080:00200000:3.0:1789760746.136591:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760746.136607:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6126, 6126] 00020000:00200000:3.0:1789760746.136615:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25092096 chunk: [25092096, 25096192) 25096192, 4096 00020000:00200000:3.0:1789760746.136620:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760746.136640:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6126, 6126] 00000008:00010000:3.0:1789760746.136658:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760746.136664:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760746.136675:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25092096, 25096192) append 0 append lockpos 0 00000080:00200000:3.0:1789760746.136688:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25092096=0x17ee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760746.136748:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760746.136751:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25096192 00000020:00010000:3.0:1789760746.136760:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760746.136763:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760746.136767:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760746.136770:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760746.136773:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760746.136897:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760746.136994:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760746.137012:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760746.137189:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad180 - object 0x240000400:5 offset 25092096<>25096192 00000100:00100000:2.0:1789760746.137682:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512645120:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760746.177236:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:3.0:1789760746.177251:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.177262:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.177319:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:3.0:1789760746.177344:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512645120:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760746.177412:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6126, 6126] 00000080:00200000:1.0:1789760746.177428:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760746.177434:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760746.177449:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25096192 i_blocks 49159 00000020:00200000:1.0:1789760746.177453:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760746.177463:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760746.177499:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25096192, count: 4096 00000080:00200000:1.0:1789760746.177506:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25096192, bytes: 4096 00000080:00200000:1.0:1789760746.177518:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760746.177527:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760746.177529:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25096192, 25100288) 0 0 00000080:00200000:1.0:1789760746.177533:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760746.177551:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6127, 6127] 00020000:00200000:1.0:1789760746.177561:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25096192 chunk: [25096192, 25100288) 25100288, 4096 00020000:00200000:1.0:1789760746.177565:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760746.177584:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6127, 6127] 00000008:00010000:1.0:1789760746.177604:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760746.177611:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760746.177623:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25096192, 25100288) append 0 append lockpos 0 00000080:00200000:1.0:1789760746.177637:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25096192=0x17ef000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760746.177698:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760746.177702:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25100288 00000020:00010000:1.0:1789760746.177711:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760746.177714:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760746.177718:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760746.177722:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760746.177725:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760746.177732:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760746.177738:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760746.177756:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760746.178048:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 25096192<>25100288 00000100:00100000:1.0:1789760746.178165:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512645248:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.223270:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973446 gen 3 00000008:00010000:1.0:1789760746.223340:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760746.223350:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.223360:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.223455:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760746.223485:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512645248:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.228142:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512645376:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760746.265763:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760746.265778:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.265804:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.265827:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760746.265856:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512645376:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.266776:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760746.266963:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512645504:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760746.269136:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760746.308310:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973448 gen 3 00000008:00010000:1.0:1789760746.308377:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760746.308398:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.308411:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.308435:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760746.308464:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512645504:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.309188:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512645632:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.350487:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973449 gen 3 00000008:00010000:1.0:1789760746.350539:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760746.350547:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.350688:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.350895:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760746.350963:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512645632:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.354376:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512645760:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.410519:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973450 gen 3 00000008:00010000:1.0:1789760746.410571:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760746.410581:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.410593:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.410614:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760746.410645:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512645760:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760746.410760:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6131, 6131] 00000080:00200000:3.0:1789760746.410785:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760746.410800:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760746.410815:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25116672 i_blocks 49159 00000020:00200000:3.0:1789760746.410818:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760746.410828:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760746.410862:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25116672, count: 4096 00000080:00200000:3.0:1789760746.410870:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25116672, bytes: 4096 00000080:00200000:3.0:1789760746.410880:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760746.410889:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760746.410892:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25116672, 25120768) 0 0 00000080:00200000:3.0:1789760746.410897:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760746.410970:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6132, 6132] 00020000:00200000:3.0:1789760746.410980:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25116672 chunk: [25116672, 25120768) 25120768, 4096 00020000:00200000:3.0:1789760746.410984:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760746.411006:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6132, 6132] 00000008:00010000:3.0:1789760746.411215:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760746.411222:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760746.411234:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25116672, 25120768) append 0 append lockpos 0 00000080:00200000:3.0:1789760746.411249:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25116672=0x17f4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760746.411308:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760746.411311:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25120768 00000020:00010000:3.0:1789760746.411320:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760746.411323:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760746.411326:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760746.411330:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760746.411333:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760746.411341:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760746.411349:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760746.411368:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760746.411559:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac380 - object 0x240000400:5 offset 25116672<>25120768 00000100:00100000:3.0:1789760746.411616:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760746.411635:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512645888:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760746.463578:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:3.0:1789760746.463593:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.463604:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.463625:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:3.0:1789760746.463656:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512645888:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.464363:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512646016:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.508302:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973452 gen 3 00000008:00010000:1.0:1789760746.508374:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760746.508383:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.508404:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.508545:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760746.508577:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512646016:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.509430:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512646144:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.540715:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973453 gen 3 00000008:00010000:1.0:1789760746.540767:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760746.540775:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.540802:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.540822:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760746.540851:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512646144:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760746.541202:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6134, 6134] 00000080:00200000:3.0:1789760746.541219:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760746.541225:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760746.541240:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25128960 i_blocks 49159 00000020:00200000:3.0:1789760746.541244:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760746.541254:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760746.541288:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25128960, count: 4096 00000080:00200000:3.0:1789760746.541295:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25128960, bytes: 4096 00000080:00200000:3.0:1789760746.541305:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760746.541314:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760746.541317:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25128960, 25133056) 0 0 00000080:00200000:3.0:1789760746.541321:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760746.541339:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6135, 6135] 00020000:00200000:3.0:1789760746.541349:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25128960 chunk: [25128960, 25133056) 25133056, 4096 00020000:00200000:3.0:1789760746.541353:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760746.541374:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6135, 6135] 00000008:00010000:3.0:1789760746.541401:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760746.541407:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760746.541419:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25128960, 25133056) append 0 append lockpos 0 00000080:00200000:3.0:1789760746.541434:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25128960=0x17f7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760746.541494:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760746.541497:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25133056 00000020:00010000:3.0:1789760746.541505:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760746.541509:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760746.541512:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760746.541515:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760746.541518:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760746.541524:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760746.541531:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760746.541548:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760746.541656:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af480 - object 0x240000400:5 offset 25128960<>25133056 00000100:00100000:2.0:1789760746.541758:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760746.541923:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512646272:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760746.575175:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973454 gen 3 00000008:00010000:3.0:1789760746.575227:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:3.0:1789760746.575235:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760746.575247:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760746.575271:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:3.0:1789760746.575300:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512646272:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.576093:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760746.576112:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512646400:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.604212:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973455 gen 3 00000008:00010000:1.0:1789760746.604265:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760746.604275:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.604287:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.604312:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760746.604345:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512646400:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.605588:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760746.605610:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512646528:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.636103:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973456 gen 3 00000008:00010000:1.0:1789760746.636156:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760746.636164:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.636177:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.636200:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760746.636229:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512646528:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.637083:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760746.637102:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512646656:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.679661:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973457 gen 3 00000008:00010000:1.0:1789760746.679716:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760746.679725:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.679739:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.679765:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760746.679799:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512646656:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760746.680559:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000008:00100000:1.0:1789760746.680673:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 25145344<>25149440 00000100:00100000:1.0:1789760746.680723:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512646784:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.728329:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973458 gen 3 00000008:00010000:1.0:1789760746.728381:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760746.728398:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.728410:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.728431:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760746.728605:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512646784:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.729325:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512646912:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.775481:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973459 gen 3 00000008:00010000:1.0:1789760746.775535:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760746.775543:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.775557:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.775586:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760746.775617:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512646912:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.776670:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760746.776703:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512647040:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.826631:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973460 gen 3 00000008:00010000:1.0:1789760746.826679:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760746.826687:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.826699:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.826721:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760746.826750:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512647040:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.827640:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512647168:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.894722:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973461 gen 3 00000008:00010000:1.0:1789760746.894779:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760746.894789:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.894801:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.894825:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760746.894858:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512647168:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.895604:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512647296:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760746.977542:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973462 gen 3 00000008:00010000:1.0:1789760746.977597:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760746.977606:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760746.977618:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760746.977640:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760746.977671:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512647296:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760746.977796:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6143, 6143] 00000080:00200000:3.0:1789760746.977811:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760746.977817:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760746.977831:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25165824 i_blocks 49159 00000020:00200000:3.0:1789760746.977835:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760746.977845:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760746.977878:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25165824, count: 4096 00000080:00200000:3.0:1789760746.977886:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25165824, bytes: 4096 00000080:00200000:3.0:1789760746.978046:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760746.978122:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760746.978125:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25165824, 25169920) 0 0 00000080:00200000:3.0:1789760746.978131:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760746.978148:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6144, 6144] 00020000:00200000:3.0:1789760746.978156:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25165824 chunk: [25165824, 25169920) 25169920, 4096 00020000:00200000:3.0:1789760746.978161:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760746.978182:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6144, 6144] 00000008:00010000:3.0:1789760746.978201:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760746.978208:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760746.978220:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25165824, 25169920) append 0 append lockpos 0 00000080:00200000:3.0:1789760746.978233:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25165824=0x1800000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760746.978294:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760746.978297:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25169920 00000020:00010000:3.0:1789760746.978305:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760746.978309:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760746.978312:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760746.978315:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760746.978318:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760746.978327:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760746.978334:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760746.978352:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760746.978641:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af480 - object 0x240000400:5 offset 25165824<>25169920 00000100:00100000:2.0:1789760746.978725:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512647424:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760747.053687:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:3.0:1789760747.053703:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760747.053716:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760747.053825:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:3.0:1789760747.053849:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512647424:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760747.054853:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512647552:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760747.102948:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973464 gen 3 00000008:00010000:1.0:1789760747.103081:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760747.103090:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.103102:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.103127:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760747.103158:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512647552:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760747.104215:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512647680:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760747.170969:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973465 gen 3 00000008:00010000:1.0:1789760747.171024:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760747.171032:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.171044:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.171069:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760747.171103:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512647680:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760747.172076:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512647808:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760747.226027:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973466 gen 3 00000008:00010000:1.0:1789760747.226080:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760747.226088:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.226101:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.226216:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760747.226248:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512647808:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760747.227162:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512647936:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760747.275119:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973467 gen 3 00000008:00010000:1.0:1789760747.275172:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760747.275180:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.275193:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.275223:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760747.275254:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512647936:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760747.275475:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6148, 6148] 00000080:00200000:3.0:1789760747.275491:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.275497:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.275512:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25186304 i_blocks 51207 00000020:00200000:3.0:1789760747.275516:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.275527:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.275571:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25186304, count: 4096 00000080:00200000:3.0:1789760747.275583:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25186304, bytes: 4096 00000080:00200000:3.0:1789760747.275596:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.275605:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.275607:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25186304, 25190400) 0 0 00000080:00200000:3.0:1789760747.275612:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.275629:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6149, 6149] 00020000:00200000:3.0:1789760747.275638:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25186304 chunk: [25186304, 25190400) 25190400, 4096 00020000:00200000:3.0:1789760747.275643:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.275662:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6149, 6149] 00000008:00010000:3.0:1789760747.275682:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.275689:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760747.275700:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25186304, 25190400) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.275715:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25186304=0x1805000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.275782:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.275784:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25190400 00000020:00010000:3.0:1789760747.275793:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.275796:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.275800:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.275803:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.275806:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.275814:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.275820:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.275839:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760747.275998:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae300 - object 0x240000400:5 offset 25186304<>25190400 00000100:00100000:3.0:1789760747.276102:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760747.276128:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512648064:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760747.293178:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760747.327409:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973468 gen 3 00000008:00010000:3.0:1789760747.327461:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:3.0:1789760747.327470:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760747.327482:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760747.327579:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:3.0:1789760747.327610:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512648064:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760747.328467:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512648192:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760747.389200:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973469 gen 3 00000008:00010000:1.0:1789760747.389264:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760747.389273:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.389286:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.389584:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760747.389617:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512648192:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760747.395031:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512648320:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760747.461748:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760747.461765:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.461831:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.461958:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760747.461992:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512648320:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760747.464708:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512648448:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760747.511975:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973471 gen 3 00000008:00010000:1.0:1789760747.512159:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760747.512169:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.512179:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.512267:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760747.512298:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512648448:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760747.513371:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512648576:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760747.550102:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973472 gen 3 00000008:00010000:1.0:1789760747.550158:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760747.550167:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760747.550178:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760747.550199:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760747.550229:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512648576:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760747.550447:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6153, 6153] 00000080:00200000:3.0:1789760747.550462:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.550468:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.550833:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25206784 i_blocks 51207 00000020:00200000:3.0:1789760747.550839:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.550850:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.550885:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25206784, count: 4096 00000080:00200000:3.0:1789760747.550932:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25206784, bytes: 4096 00000080:00200000:3.0:1789760747.550946:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.550955:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.550958:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25206784, 25210880) 0 0 00000080:00200000:3.0:1789760747.550962:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.550980:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6154, 6154] 00020000:00200000:3.0:1789760747.550988:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25206784 chunk: [25206784, 25210880) 25210880, 4096 00020000:00200000:3.0:1789760747.550991:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.551010:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6154, 6154] 00000008:00010000:3.0:1789760747.551025:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.551029:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760747.551039:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25206784, 25210880) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.551173:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25206784=0x180a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.551239:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.551242:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25210880 00000020:00010000:3.0:1789760747.551255:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.551259:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.551262:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.551266:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.551269:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.551278:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.551284:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.551309:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760747.551434:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae300 - object 0x240000400:5 offset 25206784<>25210880 00000100:00100000:2.0:1789760747.551655:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512648704:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760747.615436:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973473 gen 3 00000008:00010000:2.0:1789760747.615474:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760747.615481:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760747.615490:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760747.615507:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760747.615539:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512648704:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760747.615673:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6154, 6154] 00000080:00200000:3.0:1789760747.615686:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.615691:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.615703:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25210880 i_blocks 51207 00000020:00200000:3.0:1789760747.615707:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.615715:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.615741:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25210880, count: 4096 00000080:00200000:3.0:1789760747.615747:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25210880, bytes: 4096 00000080:00200000:3.0:1789760747.615755:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.615763:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.615765:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25210880, 25214976) 0 0 00000080:00200000:3.0:1789760747.615779:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.619937:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6155, 6155] 00020000:00200000:3.0:1789760747.619951:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25210880 chunk: [25210880, 25214976) 25214976, 4096 00020000:00200000:3.0:1789760747.619956:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.619978:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6155, 6155] 00000008:00010000:3.0:1789760747.619996:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.620001:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760747.620011:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25210880, 25214976) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.620023:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25210880=0x180b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.620090:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.620094:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25214976 00000020:00010000:3.0:1789760747.620100:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.620103:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.620106:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.620109:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.620112:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.620121:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.620128:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.620151:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760747.620231:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac000 - object 0x240000400:5 offset 25210880<>25214976 00000100:00100000:2.0:1789760747.620403:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760747.620423:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512648832:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760747.661338:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973474 gen 3 00000008:00010000:2.0:1789760747.661372:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760747.661389:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760747.661399:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760747.661416:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760747.661448:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512648832:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760747.661588:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6155, 6155] 00000080:00200000:3.0:1789760747.661600:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.661605:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.661618:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25214976 i_blocks 51207 00000020:00200000:3.0:1789760747.661621:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.661629:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.661656:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25214976, count: 4096 00000080:00200000:3.0:1789760747.661662:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25214976, bytes: 4096 00000080:00200000:3.0:1789760747.661670:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.661677:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.661679:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25214976, 25219072) 0 0 00000080:00200000:3.0:1789760747.661682:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.661696:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6156, 6156] 00020000:00200000:3.0:1789760747.661702:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25214976 chunk: [25214976, 25219072) 25219072, 4096 00020000:00200000:3.0:1789760747.661705:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.661720:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6156, 6156] 00000008:00010000:3.0:1789760747.661733:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.661737:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760747.661745:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25214976, 25219072) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.661755:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25214976=0x180c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.661812:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.661814:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25219072 00000020:00010000:3.0:1789760747.661821:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.661824:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.661827:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.661830:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.661833:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.661838:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.661844:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.661858:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760747.661967:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac700 - object 0x240000400:5 offset 25214976<>25219072 00000100:00100000:2.0:1789760747.662252:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512648960:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760747.703389:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973475 gen 3 00000008:00010000:2.0:1789760747.703427:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760747.703434:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760747.703452:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760747.703470:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760747.703493:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512648960:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760747.703637:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6156, 6156] 00000080:00200000:3.0:1789760747.703650:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.703655:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.703666:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25219072 i_blocks 51207 00000020:00200000:3.0:1789760747.703669:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.703677:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.703700:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25219072, count: 4096 00000080:00200000:3.0:1789760747.703706:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25219072, bytes: 4096 00000080:00200000:3.0:1789760747.703713:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.703720:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.703722:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25219072, 25223168) 0 0 00000080:00200000:3.0:1789760747.703725:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.703740:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6157, 6157] 00020000:00200000:3.0:1789760747.703747:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25219072 chunk: [25219072, 25223168) 25223168, 4096 00020000:00200000:3.0:1789760747.703751:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.703775:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6157, 6157] 00000008:00010000:3.0:1789760747.703790:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.703796:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760747.703805:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25219072, 25223168) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.703816:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25219072=0x180d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.703858:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.703861:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25223168 00000020:00010000:3.0:1789760747.703868:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.703871:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.703873:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.703877:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.703879:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.703885:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.703891:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.703938:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760747.704016:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad500 - object 0x240000400:5 offset 25219072<>25223168 00000100:00100000:2.0:1789760747.704196:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760747.704209:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512649088:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760747.739084:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973476 gen 3 00000008:00010000:2.0:1789760747.739121:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760747.739127:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760747.739136:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760747.739154:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760747.739176:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512649088:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760747.739384:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6157, 6157] 00000080:00200000:3.0:1789760747.739397:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.739402:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.739415:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25223168 i_blocks 51207 00000020:00200000:3.0:1789760747.739418:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.739426:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.739585:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25223168, count: 4096 00000080:00200000:3.0:1789760747.739592:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25223168, bytes: 4096 00000080:00200000:3.0:1789760747.739600:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.739607:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.739609:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25223168, 25227264) 0 0 00000080:00200000:3.0:1789760747.739613:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.739627:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6158, 6158] 00020000:00200000:3.0:1789760747.739633:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25223168 chunk: [25223168, 25227264) 25227264, 4096 00020000:00200000:3.0:1789760747.739636:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.739650:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6158, 6158] 00000008:00010000:3.0:1789760747.739665:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.739670:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760747.739679:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25223168, 25227264) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.739689:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25223168=0x180e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.739736:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.739739:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25227264 00000020:00010000:3.0:1789760747.739747:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.739751:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.739753:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.739756:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.739759:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.739766:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.739882:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.739897:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760747.740005:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 25223168<>25227264 00000100:00100000:2.0:1789760747.740278:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760747.740292:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512649216:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760747.792711:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973477 gen 3 00000008:00010000:2.0:1789760747.792745:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760747.792752:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760747.792761:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760747.792956:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:2.0:1789760747.792979:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512649216:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760747.793307:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6158, 6158] 00000080:00200000:3.0:1789760747.793319:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.793324:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.793335:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25227264 i_blocks 51207 00000020:00200000:3.0:1789760747.793338:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.793346:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.793370:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25227264, count: 4096 00000080:00200000:3.0:1789760747.793385:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25227264, bytes: 4096 00000080:00200000:3.0:1789760747.793394:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.793400:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.793402:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25227264, 25231360) 0 0 00000080:00200000:3.0:1789760747.793406:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.793420:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6159, 6159] 00020000:00200000:3.0:1789760747.793426:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25227264 chunk: [25227264, 25231360) 25231360, 4096 00020000:00200000:3.0:1789760747.793430:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.793445:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6159, 6159] 00000008:00010000:3.0:1789760747.793458:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.793463:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760747.793471:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25227264, 25231360) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.793481:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25227264=0x180f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.793524:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.793527:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25231360 00000020:00010000:3.0:1789760747.793534:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.793537:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.793540:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.793543:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.793547:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.793553:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.793557:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.793570:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760747.793730:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107afb80 - object 0x240000400:5 offset 25227264<>25231360 00000100:00100000:2.0:1789760747.793891:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512649344:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760747.829076:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760747.829087:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760747.829096:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760747.829113:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760747.829137:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512649344:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760747.830411:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6159, 6159] 00000080:00200000:3.0:1789760747.830427:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.830432:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.830444:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25231360 i_blocks 51207 00000020:00200000:3.0:1789760747.830448:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.830456:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.830482:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25231360, count: 4096 00000080:00200000:3.0:1789760747.830489:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25231360, bytes: 4096 00000080:00200000:3.0:1789760747.830500:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.830508:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.830510:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25231360, 25235456) 0 0 00000080:00200000:3.0:1789760747.830514:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.830528:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6160, 6160] 00020000:00200000:3.0:1789760747.830535:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25231360 chunk: [25231360, 25235456) 25235456, 4096 00020000:00200000:3.0:1789760747.830538:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.830554:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6160, 6160] 00000008:00010000:3.0:1789760747.830568:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.830573:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760747.830583:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25231360, 25235456) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.830595:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25231360=0x1810000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.830643:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.830645:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25235456 00000020:00010000:3.0:1789760747.832006:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.832017:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.832021:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.832025:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.832028:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.832038:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.832598:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.832625:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760747.832723:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 25231360<>25235456 00000100:00100000:2.0:1789760747.833394:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512649472:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760747.874207:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973479 gen 3 00000008:00010000:2.0:1789760747.874254:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760747.874259:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760747.874266:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760747.874280:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760747.874299:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512649472:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760747.874365:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6160, 6160] 00000080:00200000:3.0:1789760747.874386:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.874391:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.874401:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25235456 i_blocks 51207 00000020:00200000:3.0:1789760747.874405:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.874412:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.874438:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25235456, count: 4096 00000080:00200000:3.0:1789760747.874443:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25235456, bytes: 4096 00000080:00200000:3.0:1789760747.874451:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.874457:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.874459:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25235456, 25239552) 0 0 00000080:00200000:3.0:1789760747.874463:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.874486:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6161, 6161] 00020000:00200000:3.0:1789760747.874492:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25235456 chunk: [25235456, 25239552) 25239552, 4096 00020000:00200000:3.0:1789760747.874496:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.874511:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6161, 6161] 00000008:00010000:3.0:1789760747.874523:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.874528:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760747.874536:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25235456, 25239552) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.874547:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25235456=0x1811000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.874589:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.874593:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25239552 00000020:00010000:3.0:1789760747.874599:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.874603:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.874606:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.874608:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.874612:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.874618:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.874623:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.874636:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760747.874710:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 25235456<>25239552 00000100:00100000:2.0:1789760747.874790:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512649600:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760747.919720:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973480 gen 3 00000008:00010000:2.0:1789760747.919754:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760747.919760:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760747.919824:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760747.919886:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760747.919953:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512649600:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760747.922346:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6161, 6161] 00000080:00200000:3.0:1789760747.922363:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.922368:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.922391:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25239552 i_blocks 51207 00000020:00200000:3.0:1789760747.922396:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.922408:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.922433:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25239552, count: 4096 00000080:00200000:3.0:1789760747.922439:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25239552, bytes: 4096 00000080:00200000:3.0:1789760747.922450:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.922459:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.922461:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25239552, 25243648) 0 0 00000080:00200000:3.0:1789760747.922464:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.922486:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6162, 6162] 00020000:00200000:3.0:1789760747.922493:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25239552 chunk: [25239552, 25243648) 25243648, 4096 00020000:00200000:3.0:1789760747.922496:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.922513:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6162, 6162] 00000008:00010000:3.0:1789760747.922527:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.922532:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760747.922541:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25239552, 25243648) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.922551:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25239552=0x1812000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.922603:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.922606:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25243648 00000020:00010000:3.0:1789760747.922612:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.922616:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.922618:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.922622:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.922625:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.922632:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.922637:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.922658:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760747.922740:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107afb80 - object 0x240000400:5 offset 25239552<>25243648 00000100:00100000:2.0:1789760747.922997:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512649728:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760747.986267:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973481 gen 3 00000008:00010000:2.0:1789760747.986317:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760747.986323:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760747.986333:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760747.986404:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760747.986428:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512649728:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760747.986560:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6162, 6162] 00000080:00200000:3.0:1789760747.986571:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760747.986575:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760747.986585:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25243648 i_blocks 51207 00000020:00200000:3.0:1789760747.986588:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760747.986595:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760747.986672:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25243648, count: 4096 00000080:00200000:3.0:1789760747.986678:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25243648, bytes: 4096 00000080:00200000:3.0:1789760747.986684:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760747.986690:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760747.986692:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25243648, 25247744) 0 0 00000080:00200000:3.0:1789760747.986695:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760747.986706:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6163, 6163] 00020000:00200000:3.0:1789760747.986711:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25243648 chunk: [25243648, 25247744) 25247744, 4096 00020000:00200000:3.0:1789760747.986714:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760747.986727:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6163, 6163] 00000008:00010000:3.0:1789760747.986739:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760747.986744:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760747.986751:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25243648, 25247744) append 0 append lockpos 0 00000080:00200000:3.0:1789760747.986759:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25243648=0x1813000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760747.986801:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760747.986804:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25247744 00000020:00010000:3.0:1789760747.986810:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760747.986813:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.986816:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760747.986820:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760747.986823:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760747.986839:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760747.986846:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760747.986860:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760747.987150:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 25243648<>25247744 00000100:00100000:2.0:1789760747.987206:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512649856:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760748.041254:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973482 gen 3 00000008:00010000:2.0:1789760748.041293:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760748.041300:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760748.041309:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760748.041338:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:2.0:1789760748.041363:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512649856:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760748.041479:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6163, 6163] 00000080:00200000:1.0:1789760748.041493:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760748.041498:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760748.041509:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25247744 i_blocks 51207 00000020:00200000:1.0:1789760748.041513:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760748.041520:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760748.041545:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25247744, count: 4096 00000080:00200000:1.0:1789760748.041551:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25247744, bytes: 4096 00000080:00200000:1.0:1789760748.041559:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760748.041567:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760748.041569:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25247744, 25251840) 0 0 00000080:00200000:1.0:1789760748.041572:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760748.041585:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6164, 6164] 00020000:00200000:1.0:1789760748.041592:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25247744 chunk: [25247744, 25251840) 25251840, 4096 00020000:00200000:1.0:1789760748.041595:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760748.041610:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6164, 6164] 00000008:00010000:1.0:1789760748.041622:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760748.041626:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760748.041634:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25247744, 25251840) append 0 append lockpos 0 00000080:00200000:1.0:1789760748.041646:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25247744=0x1814000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760748.043298:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760748.043305:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25251840 00000020:00010000:1.0:1789760748.043316:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760748.043319:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.043323:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.043327:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760748.043330:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760748.043339:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760748.043346:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760748.043383:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760748.043467:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 25247744<>25251840 00000080:00200000:1.0:1789760748.099767:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6164, 6164] 00000080:00200000:1.0:1789760748.099780:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760748.099785:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760748.099797:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25251840 i_blocks 51207 00000020:00200000:1.0:1789760748.099800:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760748.099808:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760748.099833:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25251840, count: 4096 00000080:00200000:1.0:1789760748.099840:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25251840, bytes: 4096 00000080:00200000:1.0:1789760748.099846:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760748.099853:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760748.099855:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25251840, 25255936) 0 0 00000080:00200000:1.0:1789760748.099859:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760748.099872:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6165, 6165] 00020000:00200000:1.0:1789760748.099879:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25251840 chunk: [25251840, 25255936) 25255936, 4096 00020000:00200000:1.0:1789760748.099888:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760748.100035:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6165, 6165] 00000008:00010000:1.0:1789760748.100052:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760748.100058:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760748.100067:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25251840, 25255936) append 0 append lockpos 0 00000080:00200000:1.0:1789760748.100079:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25251840=0x1815000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760748.100121:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760748.100124:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25255936 00000020:00010000:1.0:1789760748.100132:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760748.100135:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.100138:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.100142:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760748.100144:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760748.100151:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760748.100156:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760748.100169:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760748.100239:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94700 - object 0x240000400:5 offset 25251840<>25255936 00000080:00200000:1.0:1789760748.130636:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6165, 6165] 00000080:00200000:1.0:1789760748.130647:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760748.130652:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760748.130663:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25255936 i_blocks 51207 00000020:00200000:1.0:1789760748.130666:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760748.130674:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760748.130700:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25255936, count: 4096 00000080:00200000:1.0:1789760748.130706:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25255936, bytes: 4096 00000080:00200000:1.0:1789760748.130713:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760748.130721:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760748.130723:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25255936, 25260032) 0 0 00000080:00200000:1.0:1789760748.130727:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760748.130740:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6166, 6166] 00020000:00200000:1.0:1789760748.130746:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25255936 chunk: [25255936, 25260032) 25260032, 4096 00020000:00200000:1.0:1789760748.130750:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760748.130764:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6166, 6166] 00000008:00010000:1.0:1789760748.130885:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760748.130891:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760748.130927:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25255936, 25260032) append 0 append lockpos 0 00000080:00200000:1.0:1789760748.130938:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25255936=0x1816000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760748.131112:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760748.131115:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25260032 00000020:00010000:1.0:1789760748.131123:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760748.131126:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.131130:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.131133:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760748.131136:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760748.131142:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760748.131147:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760748.131160:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760748.131229:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 25255936<>25260032 00000080:00200000:1.0:1789760748.164558:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6166, 6166] 00000080:00200000:1.0:1789760748.164570:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760748.164574:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760748.164586:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25260032 i_blocks 51207 00000020:00200000:1.0:1789760748.164589:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760748.164597:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760748.164622:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25260032, count: 4096 00000080:00200000:1.0:1789760748.164629:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25260032, bytes: 4096 00000080:00200000:1.0:1789760748.164636:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760748.164643:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760748.164645:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25260032, 25264128) 0 0 00000080:00200000:1.0:1789760748.164648:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760748.164660:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6167, 6167] 00020000:00200000:1.0:1789760748.164666:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25260032 chunk: [25260032, 25264128) 25264128, 4096 00020000:00200000:1.0:1789760748.164670:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760748.164683:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6167, 6167] 00000008:00010000:1.0:1789760748.164696:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760748.164701:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760748.164710:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25260032, 25264128) append 0 append lockpos 0 00000080:00200000:1.0:1789760748.164721:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25260032=0x1817000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760748.164775:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760748.164779:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25264128 00000020:00010000:1.0:1789760748.164786:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760748.164936:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.164940:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.164943:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760748.164945:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760748.164952:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760748.164958:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760748.164972:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760748.165056:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 25260032<>25264128 00000100:00100000:1.0:1789760748.206567:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973486 gen 3 00000008:00010000:1.0:1789760748.206603:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760748.206609:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.206617:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.206642:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760748.206666:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512650368:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760748.206771:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6167, 6167] 00000080:00200000:2.0:1789760748.206783:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760748.206891:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760748.206927:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25264128 i_blocks 51207 00000020:00200000:2.0:1789760748.206932:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760748.206939:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760748.206963:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25264128, count: 4096 00000080:00200000:2.0:1789760748.206968:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25264128, bytes: 4096 00000080:00200000:2.0:1789760748.206976:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760748.206983:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760748.206985:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25264128, 25268224) 0 0 00000080:00200000:2.0:1789760748.206989:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760748.207002:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6168, 6168] 00020000:00200000:2.0:1789760748.207008:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25264128 chunk: [25264128, 25268224) 25268224, 4096 00020000:00200000:2.0:1789760748.207011:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760748.207026:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6168, 6168] 00000008:00010000:2.0:1789760748.207039:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760748.207102:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760748.207112:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25264128, 25268224) append 0 append lockpos 0 00000080:00200000:2.0:1789760748.207122:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25264128=0x1818000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760748.207162:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760748.207165:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25268224 00000020:00010000:2.0:1789760748.207171:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760748.207174:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.207177:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.207180:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760748.207183:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760748.207189:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760748.207193:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760748.207206:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760748.207275:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 25264128<>25268224 00000100:00100000:3.0:1789760748.207347:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512650496:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760748.259104:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:2.0:1789760748.259115:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760748.259124:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760748.259149:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:2.0:1789760748.259172:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512650496:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760748.259316:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6168, 6168] 00000080:00200000:1.0:1789760748.259328:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760748.259333:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760748.259344:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25268224 i_blocks 51207 00000020:00200000:1.0:1789760748.259347:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760748.259355:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760748.259391:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25268224, count: 4096 00000080:00200000:1.0:1789760748.259397:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25268224, bytes: 4096 00000080:00200000:1.0:1789760748.259405:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760748.259412:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760748.259414:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25268224, 25272320) 0 0 00000080:00200000:1.0:1789760748.259418:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760748.259431:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6169, 6169] 00020000:00200000:1.0:1789760748.259437:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25268224 chunk: [25268224, 25272320) 25272320, 4096 00020000:00200000:1.0:1789760748.259440:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760748.259455:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6169, 6169] 00000008:00010000:1.0:1789760748.259469:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760748.259474:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760748.259483:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25268224, 25272320) append 0 append lockpos 0 00000080:00200000:1.0:1789760748.259494:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25268224=0x1819000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760748.259540:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760748.259543:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25272320 00000020:00010000:1.0:1789760748.259549:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760748.259552:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.259554:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.259557:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760748.259560:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760748.259566:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760748.259571:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760748.259585:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760748.259660:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 25268224<>25272320 00000100:00100000:1.0:1789760748.259702:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760748.259715:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512650624:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760748.300836:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973488 gen 3 00000008:00010000:1.0:1789760748.301045:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760748.301052:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.301059:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.301082:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760748.301103:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512650624:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760748.301293:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6169, 6169] 00000080:00200000:2.0:1789760748.301305:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760748.301310:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760748.301320:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25272320 i_blocks 51207 00000020:00200000:2.0:1789760748.301324:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760748.301333:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760748.301357:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25272320, count: 4096 00000080:00200000:2.0:1789760748.301362:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25272320, bytes: 4096 00000080:00200000:2.0:1789760748.301370:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760748.301387:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760748.301389:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25272320, 25276416) 0 0 00000080:00200000:2.0:1789760748.301392:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760748.301404:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6170, 6170] 00020000:00200000:2.0:1789760748.301411:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25272320 chunk: [25272320, 25276416) 25276416, 4096 00020000:00200000:2.0:1789760748.301415:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760748.301428:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6170, 6170] 00000008:00010000:2.0:1789760748.301442:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760748.301446:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760748.301454:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25272320, 25276416) append 0 append lockpos 0 00000080:00200000:2.0:1789760748.301464:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25272320=0x181a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760748.301503:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760748.301506:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25276416 00000020:00010000:2.0:1789760748.301512:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760748.301516:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.301519:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.301522:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760748.301524:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760748.301531:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760748.301537:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760748.301551:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760748.301623:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 25272320<>25276416 00000100:00100000:3.0:1789760748.301684:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512650752:192.168.203.156@tcp:4:dd.0 00010000:00010000:3.0:1789760748.319547:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760748.348323:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973489 gen 3 00000008:00010000:2.0:1789760748.348356:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:2.0:1789760748.348362:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760748.348382:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760748.348402:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:2.0:1789760748.348424:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512650752:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760748.348512:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6170, 6170] 00000080:00200000:1.0:1789760748.348524:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760748.348529:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760748.348540:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25276416 i_blocks 51207 00000020:00200000:1.0:1789760748.348543:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760748.348551:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760748.348579:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25276416, count: 4096 00000080:00200000:1.0:1789760748.348586:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25276416, bytes: 4096 00000080:00200000:1.0:1789760748.348594:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760748.348600:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760748.348602:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25276416, 25280512) 0 0 00000080:00200000:1.0:1789760748.348606:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760748.348619:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6171, 6171] 00020000:00200000:1.0:1789760748.348626:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25276416 chunk: [25276416, 25280512) 25280512, 4096 00020000:00200000:1.0:1789760748.348630:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760748.348655:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6171, 6171] 00000008:00010000:1.0:1789760748.348668:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760748.348672:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760748.348681:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25276416, 25280512) append 0 append lockpos 0 00000080:00200000:1.0:1789760748.348691:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25276416=0x181b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760748.348732:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760748.348734:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25280512 00000020:00010000:1.0:1789760748.348741:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760748.348744:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.348747:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.348749:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760748.348752:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760748.348757:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760748.348771:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760748.348784:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760748.348859:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 25276416<>25280512 00000100:00100000:1.0:1789760748.386190:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973490 gen 3 00000008:00010000:1.0:1789760748.386230:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760748.386236:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.386244:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.386267:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760748.386290:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512650880:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760748.386454:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6171, 6171] 00000080:00200000:2.0:1789760748.386467:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760748.386472:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760748.386483:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25280512 i_blocks 51207 00000020:00200000:2.0:1789760748.386486:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760748.386494:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760748.386520:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25280512, count: 4096 00000080:00200000:2.0:1789760748.386526:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25280512, bytes: 4096 00000080:00200000:2.0:1789760748.386535:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760748.386542:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760748.386545:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25280512, 25284608) 0 0 00000080:00200000:2.0:1789760748.386549:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760748.386562:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6172, 6172] 00020000:00200000:2.0:1789760748.386568:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25280512 chunk: [25280512, 25284608) 25284608, 4096 00020000:00200000:2.0:1789760748.386572:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760748.386587:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6172, 6172] 00000008:00010000:2.0:1789760748.386600:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760748.386604:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760748.386614:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25280512, 25284608) append 0 append lockpos 0 00000080:00200000:2.0:1789760748.386624:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25280512=0x181c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760748.386672:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760748.386675:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25284608 00000020:00010000:2.0:1789760748.386683:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760748.386686:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.386688:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.386692:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760748.386694:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760748.386701:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760748.386707:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760748.386720:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760748.386810:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 25280512<>25284608 00000100:00100000:2.0:1789760748.386844:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760748.386857:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512651008:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760748.418609:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:2.0:1789760748.418621:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760748.418630:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760748.418653:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:2.0:1789760748.418679:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512651008:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760748.418777:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6172, 6172] 00000080:00200000:1.0:1789760748.418797:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760748.418804:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760748.418816:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25284608 i_blocks 51207 00000020:00200000:1.0:1789760748.418819:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760748.418828:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760748.418856:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25284608, count: 4096 00000080:00200000:1.0:1789760748.418862:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25284608, bytes: 4096 00000080:00200000:1.0:1789760748.418870:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760748.418877:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760748.418879:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25284608, 25288704) 0 0 00000080:00200000:1.0:1789760748.418882:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760748.418895:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6173, 6173] 00020000:00200000:1.0:1789760748.418928:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25284608 chunk: [25284608, 25288704) 25288704, 4096 00020000:00200000:1.0:1789760748.418932:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760748.418949:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6173, 6173] 00000008:00010000:1.0:1789760748.418962:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760748.418967:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760748.418976:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25284608, 25288704) append 0 append lockpos 0 00000080:00200000:1.0:1789760748.418987:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25284608=0x181d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760748.419029:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760748.419032:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25288704 00000020:00010000:1.0:1789760748.419038:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760748.419053:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.419057:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.419061:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760748.419064:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760748.419070:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760748.419076:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760748.419089:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760748.419164:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 25284608<>25288704 00000100:00100000:1.0:1789760748.419214:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760748.419228:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512651136:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760748.468441:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973492 gen 3 00000008:00010000:1.0:1789760748.468485:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760748.468491:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.468498:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.468523:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760748.468546:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512651136:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760748.469166:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512651264:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760748.556147:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973493 gen 3 00000008:00010000:1.0:1789760748.556181:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760748.556187:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.556198:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.556220:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760748.556243:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512651264:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760748.556338:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6174, 6174] 00000080:00200000:2.0:1789760748.556350:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760748.556356:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760748.556367:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25292800 i_blocks 51207 00000020:00200000:2.0:1789760748.556381:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760748.556389:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760748.556413:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25292800, count: 4096 00000080:00200000:2.0:1789760748.556419:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25292800, bytes: 4096 00000080:00200000:2.0:1789760748.556427:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760748.556434:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760748.556436:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25292800, 25296896) 0 0 00000080:00200000:2.0:1789760748.556440:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760748.556454:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6175, 6175] 00020000:00200000:2.0:1789760748.556461:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25292800 chunk: [25292800, 25296896) 25296896, 4096 00020000:00200000:2.0:1789760748.556465:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760748.556481:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6175, 6175] 00000008:00010000:2.0:1789760748.556494:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760748.556499:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760748.556508:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25292800, 25296896) append 0 append lockpos 0 00000080:00200000:2.0:1789760748.556518:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25292800=0x181f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760748.556563:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760748.556567:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25296896 00000020:00010000:2.0:1789760748.556574:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760748.556578:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.556581:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.556584:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760748.556587:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760748.556594:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760748.556600:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760748.556614:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760748.556691:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 25292800<>25296896 00000100:00100000:3.0:1789760748.556925:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760748.556939:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512651392:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760748.610422:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973494 gen 3 00000008:00010000:2.0:1789760748.610457:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:2.0:1789760748.610464:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760748.610472:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760748.610493:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:2.0:1789760748.610515:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512651392:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760748.610627:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6175, 6175] 00000080:00200000:1.0:1789760748.610640:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760748.610645:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760748.610665:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25296896 i_blocks 51207 00000020:00200000:1.0:1789760748.610669:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760748.610681:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760748.610708:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25296896, count: 4096 00000080:00200000:1.0:1789760748.610714:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25296896, bytes: 4096 00000080:00200000:1.0:1789760748.610728:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760748.610735:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760748.610737:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25296896, 25300992) 0 0 00000080:00200000:1.0:1789760748.610740:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760748.610753:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6176, 6176] 00020000:00200000:1.0:1789760748.610760:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25296896 chunk: [25296896, 25300992) 25300992, 4096 00020000:00200000:1.0:1789760748.610763:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760748.610778:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6176, 6176] 00000008:00010000:1.0:1789760748.610791:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760748.610795:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760748.610804:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25296896, 25300992) append 0 append lockpos 0 00000080:00200000:1.0:1789760748.610814:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25296896=0x1820000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760748.610856:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760748.610859:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25300992 00000020:00010000:1.0:1789760748.610866:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760748.610869:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.610872:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.610875:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760748.610877:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760748.610883:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760748.610888:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760748.610937:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760748.611021:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 25296896<>25300992 00000100:00100000:1.0:1789760748.611047:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760748.611059:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512651520:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760748.684765:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973495 gen 3 00000008:00010000:1.0:1789760748.684803:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760748.684810:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.684819:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.684897:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760748.684951:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512651520:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760748.686679:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6176, 6176] 00000080:00200000:2.0:1789760748.686697:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760748.686703:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760748.686742:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25300992 i_blocks 51207 00000020:00200000:2.0:1789760748.686746:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760748.686754:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760748.686818:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25300992, count: 4096 00000080:00200000:2.0:1789760748.686825:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25300992, bytes: 4096 00000080:00200000:2.0:1789760748.686834:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760748.686841:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760748.686843:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25300992, 25305088) 0 0 00000080:00200000:2.0:1789760748.686848:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760748.686860:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6177, 6177] 00020000:00200000:2.0:1789760748.686868:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25300992 chunk: [25300992, 25305088) 25305088, 4096 00020000:00200000:2.0:1789760748.686871:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760748.686885:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6177, 6177] 00000008:00010000:2.0:1789760748.686914:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760748.686920:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760748.686929:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25300992, 25305088) append 0 append lockpos 0 00000080:00200000:2.0:1789760748.686941:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25300992=0x1821000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760748.686987:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760748.686990:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25305088 00000020:00010000:2.0:1789760748.686997:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760748.687000:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.687003:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.687006:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760748.687009:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760748.687015:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760748.687021:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760748.687036:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760748.687121:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 25300992<>25305088 00000100:00100000:3.0:1789760748.687223:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512651648:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760748.760248:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:2.0:1789760748.760260:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760748.760269:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760748.760290:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:2.0:1789760748.760312:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512651648:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760748.760406:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6177, 6177] 00000080:00200000:1.0:1789760748.760417:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760748.760423:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760748.760434:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25305088 i_blocks 51207 00000020:00200000:1.0:1789760748.760438:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760748.760447:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760748.760473:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25305088, count: 4096 00000080:00200000:1.0:1789760748.760479:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25305088, bytes: 4096 00000080:00200000:1.0:1789760748.760488:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760748.760495:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760748.760497:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25305088, 25309184) 0 0 00000080:00200000:1.0:1789760748.760501:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760748.760515:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6178, 6178] 00020000:00200000:1.0:1789760748.760522:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25305088 chunk: [25305088, 25309184) 25309184, 4096 00020000:00200000:1.0:1789760748.760526:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760748.760540:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6178, 6178] 00000008:00010000:1.0:1789760748.760554:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760748.760560:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760748.760570:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25305088, 25309184) append 0 append lockpos 0 00000080:00200000:1.0:1789760748.760581:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25305088=0x1822000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760748.760625:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760748.760628:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25309184 00000020:00010000:1.0:1789760748.760634:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760748.760637:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.760640:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.760643:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760748.760646:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760748.760652:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760748.760658:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760748.760671:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760748.760750:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 25305088<>25309184 00000080:00200000:1.0:1789760748.814935:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6178, 6178] 00000080:00200000:1.0:1789760748.814948:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760748.814953:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760748.814966:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25309184 i_blocks 51207 00000020:00200000:1.0:1789760748.814969:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760748.814977:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760748.815006:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25309184, count: 4096 00000080:00200000:1.0:1789760748.815012:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25309184, bytes: 4096 00000080:00200000:1.0:1789760748.815020:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760748.815028:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760748.815030:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25309184, 25313280) 0 0 00000080:00200000:1.0:1789760748.815033:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760748.815048:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6179, 6179] 00020000:00200000:1.0:1789760748.815056:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25309184 chunk: [25309184, 25313280) 25313280, 4096 00020000:00200000:1.0:1789760748.815060:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760748.815075:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6179, 6179] 00000008:00010000:1.0:1789760748.815089:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760748.815094:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760748.815103:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25309184, 25313280) append 0 append lockpos 0 00000080:00200000:1.0:1789760748.815113:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25309184=0x1823000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760748.815154:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760748.815157:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25313280 00000020:00010000:1.0:1789760748.815163:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760748.815166:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.815169:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760748.815171:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760748.815174:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760748.815181:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760748.815186:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760748.815198:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760748.815280:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94380 - object 0x240000400:5 offset 25309184<>25313280 00000100:00100000:1.0:1789760748.815321:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760748.815339:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512651904:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760748.880247:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973497 gen 3 00000100:00100000:1.0:1789760748.880279:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005c94380 x1876699512651904/t4294973498(4294973498) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760764 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760748.880319:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760748.880325:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760748.880334:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760748.880362:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760748.880399:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512651904:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760748.880936:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6179, 6179] 00000080:00200000:2.0:1789760748.880949:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760748.880954:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760748.880966:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25313280 i_blocks 51207 00000020:00200000:2.0:1789760748.880970:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760748.880978:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760748.881002:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25313280, count: 4096 00000080:00200000:2.0:1789760748.881008:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25313280, bytes: 4096 00000080:00200000:2.0:1789760748.881018:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760748.881025:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760748.881027:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25313280, 25317376) 0 0 00000080:00200000:2.0:1789760748.881031:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760748.881044:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6180, 6180] 00020000:00200000:2.0:1789760748.881050:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25313280 chunk: [25313280, 25317376) 25317376, 4096 00020000:00200000:2.0:1789760748.881054:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760748.881070:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6180, 6180] 00000008:00010000:2.0:1789760748.881082:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760748.881087:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760748.881096:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25313280, 25317376) append 0 append lockpos 0 00000080:00200000:2.0:1789760748.881106:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25313280=0x1824000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760748.881149:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760748.881152:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25317376 00000020:00010000:2.0:1789760748.881170:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760748.881174:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.881178:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.881181:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760748.881184:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760748.881192:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760748.881197:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760748.881211:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760748.881284:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 25313280<>25317376 00000100:00100000:3.0:1789760748.881354:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512652032:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760748.917062:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973499 gen 3 00000008:00010000:3.0:1789760748.917113:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760748.917120:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760748.917127:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760748.917304:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760748.917329:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512652032:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760748.918562:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6180, 6180] 00000080:00200000:2.0:1789760748.918578:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760748.918584:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760748.918597:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25317376 i_blocks 51207 00000020:00200000:2.0:1789760748.918601:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760748.918610:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760748.919104:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25317376, count: 4096 00000080:00200000:2.0:1789760748.919112:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25317376, bytes: 4096 00000080:00200000:2.0:1789760748.919124:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760748.919132:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760748.919134:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25317376, 25321472) 0 0 00000080:00200000:2.0:1789760748.919137:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760748.919231:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6181, 6181] 00020000:00200000:2.0:1789760748.919238:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25317376 chunk: [25317376, 25321472) 25321472, 4096 00020000:00200000:2.0:1789760748.919242:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760748.919259:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6181, 6181] 00000008:00010000:2.0:1789760748.919273:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760748.919278:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760748.919287:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25317376, 25321472) append 0 append lockpos 0 00000080:00200000:2.0:1789760748.919300:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25317376=0x1825000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760748.919353:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760748.919357:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25321472 00000020:00010000:2.0:1789760748.919363:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760748.919377:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.919380:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.919383:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760748.919385:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760748.919392:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760748.919398:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760748.919419:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760748.919501:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 25317376<>25321472 00000100:00100000:3.0:1789760748.919582:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512652160:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760748.949630:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973500 gen 3 00000008:00010000:3.0:1789760748.949664:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760748.949672:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760748.949680:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760748.949698:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760748.949723:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512652160:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760748.949931:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6181, 6181] 00000080:00200000:2.0:1789760748.949948:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760748.949953:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760748.949965:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25321472 i_blocks 51207 00000020:00200000:2.0:1789760748.949969:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760748.949977:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760748.950004:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25321472, count: 4096 00000080:00200000:2.0:1789760748.950011:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25321472, bytes: 4096 00000080:00200000:2.0:1789760748.950018:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760748.950025:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760748.950027:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25321472, 25325568) 0 0 00000080:00200000:2.0:1789760748.950031:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760748.950044:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6182, 6182] 00020000:00200000:2.0:1789760748.950052:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25321472 chunk: [25321472, 25325568) 25325568, 4096 00020000:00200000:2.0:1789760748.950055:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760748.950070:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6182, 6182] 00000008:00010000:2.0:1789760748.950083:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760748.950087:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760748.950096:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25321472, 25325568) append 0 append lockpos 0 00000080:00200000:2.0:1789760748.950107:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25321472=0x1826000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760748.950155:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760748.950157:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25325568 00000020:00010000:2.0:1789760748.950164:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760748.950168:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.950170:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.950174:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760748.950176:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760748.950182:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760748.950189:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760748.950202:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000008:00100000:2.0:1789760748.950284:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 25321472<>25325568 00000100:00100000:3.0:1789760748.950380:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760748.950394:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512652288:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760748.979724:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973501 gen 3 00000008:00010000:3.0:1789760748.979757:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760748.979764:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760748.979772:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760748.979829:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760748.979860:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512652288:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760748.980113:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6182, 6182] 00000080:00200000:2.0:1789760748.980127:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760748.980132:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760748.980143:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25325568 i_blocks 51207 00000020:00200000:2.0:1789760748.980146:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760748.980154:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760748.980178:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25325568, count: 4096 00000080:00200000:2.0:1789760748.980183:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25325568, bytes: 4096 00000080:00200000:2.0:1789760748.980191:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760748.980199:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760748.980201:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25325568, 25329664) 0 0 00000080:00200000:2.0:1789760748.980204:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760748.980218:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6183, 6183] 00020000:00200000:2.0:1789760748.980225:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25325568 chunk: [25325568, 25329664) 25329664, 4096 00020000:00200000:2.0:1789760748.980228:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760748.980243:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6183, 6183] 00000008:00010000:2.0:1789760748.980255:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760748.980260:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760748.980269:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25325568, 25329664) append 0 append lockpos 0 00000080:00200000:2.0:1789760748.980280:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25325568=0x1827000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760748.980322:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760748.980325:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25329664 00000020:00010000:2.0:1789760748.980332:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760748.980335:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.980338:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760748.980341:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760748.980344:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760748.980350:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760748.980355:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760748.980382:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760748.980456:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 25325568<>25329664 00000100:00100000:3.0:1789760748.980510:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512652416:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760749.024572:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760749.024585:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.024594:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.024612:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760749.024636:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512652416:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760749.024715:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6183, 6183] 00000080:00200000:2.0:1789760749.024728:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.024733:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.024746:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25329664 i_blocks 51207 00000020:00200000:2.0:1789760749.024749:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.024773:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.024811:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25329664, count: 4096 00000080:00200000:2.0:1789760749.024817:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25329664, bytes: 4096 00000080:00200000:2.0:1789760749.024825:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.024832:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.024834:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25329664, 25333760) 0 0 00000080:00200000:2.0:1789760749.024838:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.024852:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6184, 6184] 00020000:00200000:2.0:1789760749.024858:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25329664 chunk: [25329664, 25333760) 25333760, 4096 00020000:00200000:2.0:1789760749.024861:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.024877:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6184, 6184] 00000008:00010000:2.0:1789760749.024890:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.024895:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760749.024943:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25329664, 25333760) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.024955:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25329664=0x1828000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.024999:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.025002:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25333760 00000020:00010000:2.0:1789760749.025009:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.025012:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.025015:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.025018:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.025020:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.025027:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.025032:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.025055:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760749.025127:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 25329664<>25333760 00000100:00100000:3.0:1789760749.025473:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512652544:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760749.062471:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973502 gen 3 00000100:00100000:3.0:1789760749.062498:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368e680 x1876699512652544/t4294973503(4294973503) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760765 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760749.062534:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760749.062540:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.062547:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.062564:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760749.062585:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512652544:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760749.062665:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6184, 6184] 00000080:00200000:2.0:1789760749.062678:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.062684:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.062694:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25333760 i_blocks 51207 00000020:00200000:2.0:1789760749.062697:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.062705:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.062736:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25333760, count: 4096 00000080:00200000:2.0:1789760749.062742:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25333760, bytes: 4096 00000080:00200000:2.0:1789760749.062751:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.062769:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.062771:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25333760, 25337856) 0 0 00000080:00200000:2.0:1789760749.062776:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.062796:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6185, 6185] 00020000:00200000:2.0:1789760749.062803:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25333760 chunk: [25333760, 25337856) 25337856, 4096 00020000:00200000:2.0:1789760749.062807:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.062824:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6185, 6185] 00000008:00010000:2.0:1789760749.062843:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.062848:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760749.062858:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25333760, 25337856) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.062870:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25333760=0x1829000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.063186:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.063189:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25337856 00000020:00010000:2.0:1789760749.063197:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.063201:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.063204:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.063208:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.063211:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.063218:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.063224:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.063238:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000008:00100000:2.0:1789760749.063324:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 25333760<>25337856 00000100:00100000:2.0:1789760749.063359:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760749.063382:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512652672:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760749.101527:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973504 gen 3 00000008:00010000:3.0:1789760749.101572:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760749.101578:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.101586:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.101604:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760749.101626:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512652672:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760749.101703:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6185, 6185] 00000080:00200000:2.0:1789760749.101715:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.101721:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.101732:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25337856 i_blocks 51207 00000020:00200000:2.0:1789760749.101736:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.101743:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.101832:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25337856, count: 4096 00000080:00200000:2.0:1789760749.101839:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25337856, bytes: 4096 00000080:00200000:2.0:1789760749.101848:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.101855:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.101857:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25337856, 25341952) 0 0 00000080:00200000:2.0:1789760749.101862:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.101874:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6186, 6186] 00020000:00200000:2.0:1789760749.101881:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25337856 chunk: [25337856, 25341952) 25341952, 4096 00020000:00200000:2.0:1789760749.101885:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.101988:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6186, 6186] 00000008:00010000:2.0:1789760749.102002:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.102007:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760749.102015:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25337856, 25341952) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.102026:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25337856=0x182a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.102170:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.102173:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25341952 00000020:00010000:2.0:1789760749.102180:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.102183:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.102186:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.102189:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.102192:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.102199:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.102205:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.102217:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000008:00100000:2.0:1789760749.102286:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 25337856<>25341952 00000100:00100000:2.0:1789760749.102321:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760749.102333:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512652800:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760749.143403:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973505 gen 3 00000008:00010000:3.0:1789760749.143439:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760749.143445:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.143453:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.143471:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760749.143492:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512652800:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760749.143689:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6186, 6186] 00000080:00200000:2.0:1789760749.143703:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.143708:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.143719:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25341952 i_blocks 51207 00000020:00200000:2.0:1789760749.143721:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.143729:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.143753:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25341952, count: 4096 00000080:00200000:2.0:1789760749.143768:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25341952, bytes: 4096 00000080:00200000:2.0:1789760749.143776:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.143782:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.143784:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25341952, 25346048) 0 0 00000080:00200000:2.0:1789760749.143793:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.143805:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6187, 6187] 00020000:00200000:2.0:1789760749.143811:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25341952 chunk: [25341952, 25346048) 25346048, 4096 00020000:00200000:2.0:1789760749.143814:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.143828:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6187, 6187] 00000008:00010000:2.0:1789760749.143841:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.143846:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760749.143854:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25341952, 25346048) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.143864:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25341952=0x182b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.143931:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.143934:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25346048 00000020:00010000:2.0:1789760749.143940:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.143943:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.143946:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.143950:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.143953:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.143959:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.143965:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.143980:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760749.144234:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 25341952<>25346048 00000100:00100000:3.0:1789760749.144292:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512652928:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760749.213945:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973506 gen 3 00000008:00010000:3.0:1789760749.213995:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760749.214003:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.214012:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.214345:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6187, 6187] 00000080:00200000:2.0:1789760749.214360:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.214376:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.214389:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25346048 i_blocks 51207 00000020:00200000:2.0:1789760749.214394:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.214403:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.214431:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25346048, count: 4096 00000080:00200000:2.0:1789760749.214437:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25346048, bytes: 4096 00000080:00200000:2.0:1789760749.214448:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.214455:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.214457:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25346048, 25350144) 0 0 00000080:00200000:2.0:1789760749.214461:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.214474:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6188, 6188] 00020000:00200000:2.0:1789760749.214480:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25346048 chunk: [25346048, 25350144) 25350144, 4096 00020000:00200000:2.0:1789760749.214484:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.214500:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6188, 6188] 00000008:00010000:2.0:1789760749.214513:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.214518:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760749.214527:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25346048, 25350144) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.214538:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25346048=0x182c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.214582:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.214584:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25350144 00000020:00010000:2.0:1789760749.214590:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.214594:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.214597:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.214601:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.214603:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.214610:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.214616:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.214629:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00010000:3.0:1789760749.215137:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760749.215158:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512652928:192.168.203.156@tcp:4:dd.0 00000008:00100000:2.0:1789760749.215329:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 25346048<>25350144 00000100:00100000:3.0:1789760749.215392:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760749.215467:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512653056:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760749.272137:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973507 gen 3 00000008:00010000:3.0:1789760749.272172:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760749.272178:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.272186:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.272209:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760749.272231:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512653056:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760749.272313:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6188, 6188] 00000080:00200000:1.0:1789760749.272326:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760749.272331:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760749.272342:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25350144 i_blocks 51207 00000020:00200000:1.0:1789760749.272345:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760749.272353:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760749.272386:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25350144, count: 4096 00000080:00200000:1.0:1789760749.272392:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25350144, bytes: 4096 00000080:00200000:1.0:1789760749.272399:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760749.272405:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760749.272407:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25350144, 25354240) 0 0 00000080:00200000:1.0:1789760749.272411:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760749.272423:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6189, 6189] 00020000:00200000:1.0:1789760749.272429:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25350144 chunk: [25350144, 25354240) 25354240, 4096 00020000:00200000:1.0:1789760749.272432:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760749.272446:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6189, 6189] 00000008:00010000:1.0:1789760749.272458:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760749.272462:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760749.272470:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25350144, 25354240) append 0 append lockpos 0 00000080:00200000:1.0:1789760749.272479:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25350144=0x182d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760749.272517:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760749.272520:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25354240 00000020:00010000:1.0:1789760749.272526:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760749.272529:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760749.272532:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760749.272535:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760749.272537:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760749.272543:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760749.272549:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760749.272561:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760749.272630:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 25350144<>25354240 00000080:00200000:1.0:1789760749.317061:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6189, 6189] 00000080:00200000:1.0:1789760749.317074:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760749.317079:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760749.317090:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25354240 i_blocks 51207 00000020:00200000:1.0:1789760749.317094:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760749.317101:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760749.317125:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25354240, count: 4096 00000080:00200000:1.0:1789760749.317130:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25354240, bytes: 4096 00000080:00200000:1.0:1789760749.317138:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760749.317145:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760749.317147:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25354240, 25358336) 0 0 00000080:00200000:1.0:1789760749.317150:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760749.317162:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6190, 6190] 00020000:00200000:1.0:1789760749.317170:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25354240 chunk: [25354240, 25358336) 25358336, 4096 00020000:00200000:1.0:1789760749.317174:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760749.317190:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6190, 6190] 00000008:00010000:1.0:1789760749.317205:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760749.317209:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760749.317217:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25354240, 25358336) append 0 append lockpos 0 00000080:00200000:1.0:1789760749.317227:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25354240=0x182e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760749.317271:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760749.317274:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25358336 00000020:00010000:1.0:1789760749.317280:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760749.317284:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760749.317286:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760749.317290:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760749.317292:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760749.317298:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760749.317304:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760749.317318:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760749.317538:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 25354240<>25358336 00000100:00100000:1.0:1789760749.317581:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760749.317595:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512653312:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760749.341291:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760749.341765:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e6cf7267-9b19-462a-a263-14115df15dab:2361:1876699512653440:192.168.203.156@tcp:400:kworker.0 00000100:00100000:2.0:1789760749.347880:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760749.347913:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512653568:192.168.203.156@tcp:400:kworker.0 00000100:00100000:2.0:1789760749.351213:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760749.351237:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2360:1876699512653696:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760749.354300:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512653568:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760749.354318:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e6cf7267-9b19-462a-a263-14115df15dab:2361:1876699512653440:192.168.203.156@tcp:400:kworker.0 00000100:00100000:2.0:1789760749.358171:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2360:1876699512653696:192.168.203.156@tcp:400:kworker.0 00000008:00010000:1.0:1789760749.367205:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760749.367217:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760749.367226:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760749.367246:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760749.367271:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512653312:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760749.367350:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6190, 6190] 00000080:00200000:3.0:1789760749.367447:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760749.367453:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760749.367464:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25358336 i_blocks 51207 00000020:00200000:3.0:1789760749.367467:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760749.367475:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760749.367501:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25358336, count: 4096 00000080:00200000:3.0:1789760749.367507:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25358336, bytes: 4096 00000080:00200000:3.0:1789760749.367515:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760749.367522:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760749.367524:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25358336, 25362432) 0 0 00000080:00200000:3.0:1789760749.367527:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760749.367538:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6191, 6191] 00020000:00200000:3.0:1789760749.367545:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25358336 chunk: [25358336, 25362432) 25362432, 4096 00020000:00200000:3.0:1789760749.367549:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760749.367564:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6191, 6191] 00000008:00010000:3.0:1789760749.367578:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760749.367583:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760749.367592:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25358336, 25362432) append 0 append lockpos 0 00000080:00200000:3.0:1789760749.367602:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25358336=0x182f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760749.367647:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760749.367650:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25362432 00000020:00010000:3.0:1789760749.367657:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760749.367660:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760749.367663:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760749.367666:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760749.367668:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760749.367674:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760749.367680:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760749.367695:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760749.367785:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad500 - object 0x240000400:5 offset 25358336<>25362432 00000100:00100000:2.0:1789760749.368203:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512653824:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760749.397684:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973510 gen 3 00000008:00010000:3.0:1789760749.397727:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760749.397734:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.397742:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.397889:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760749.397933:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512653824:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760749.398130:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6191, 6191] 00000080:00200000:2.0:1789760749.398142:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.398147:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.398158:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25362432 i_blocks 51207 00000020:00200000:2.0:1789760749.398162:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.398169:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.398196:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25362432, count: 4096 00000080:00200000:2.0:1789760749.398202:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25362432, bytes: 4096 00000080:00200000:2.0:1789760749.398209:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.398216:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.398218:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25362432, 25366528) 0 0 00000080:00200000:2.0:1789760749.398222:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.398235:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6192, 6192] 00020000:00200000:2.0:1789760749.398243:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25362432 chunk: [25362432, 25366528) 25366528, 4096 00020000:00200000:2.0:1789760749.398247:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.398262:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6192, 6192] 00000008:00010000:2.0:1789760749.398275:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.398280:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760749.398289:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25362432, 25366528) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.398300:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25362432=0x1830000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.398342:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.398345:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25366528 00000020:00010000:2.0:1789760749.398351:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.398688:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.398692:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.398695:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.398697:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.398705:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.398712:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.398731:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000008:00100000:2.0:1789760749.398977:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 25362432<>25366528 00000100:00100000:3.0:1789760749.399082:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512653952:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760749.440696:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760749.440707:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.440716:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.440734:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000080:00200000:2.0:1789760749.440992:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6192, 6192] 00000080:00200000:2.0:1789760749.441007:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.441012:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.441023:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25366528 i_blocks 51207 00000020:00200000:2.0:1789760749.441027:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.441045:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.441072:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25366528, count: 4096 00000080:00200000:2.0:1789760749.441079:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25366528, bytes: 4096 00000080:00200000:2.0:1789760749.441087:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.441094:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.441096:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25366528, 25370624) 0 0 00000080:00200000:2.0:1789760749.441100:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.441112:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6193, 6193] 00020000:00200000:2.0:1789760749.441120:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25366528 chunk: [25366528, 25370624) 25370624, 4096 00020000:00200000:2.0:1789760749.441124:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.441137:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6193, 6193] 00000008:00010000:2.0:1789760749.441151:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.441156:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760749.441165:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25366528, 25370624) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.441175:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25366528=0x1831000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.441214:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.441217:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25370624 00000020:00010000:2.0:1789760749.441223:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.441226:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.441229:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.441232:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.441235:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.441241:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.441247:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1789760749.441407:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512653952:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760749.441436:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:3.0:1789760749.441511:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad500 - object 0x240000400:5 offset 25366528<>25370624 00000100:00100000:3.0:1789760749.441547:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760749.441558:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512654080:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760749.473668:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973511 gen 3 00000100:00100000:3.0:1789760749.473699:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc0107ad500 x1876699512654080/t4294973512(4294973512) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760765 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760749.473735:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760749.473741:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.473750:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.473776:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760749.473797:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512654080:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760749.473875:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6193, 6193] 00000080:00200000:2.0:1789760749.473887:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.473892:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.473936:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25370624 i_blocks 51207 00000020:00200000:2.0:1789760749.473940:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.473948:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.473974:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25370624, count: 4096 00000080:00200000:2.0:1789760749.473980:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25370624, bytes: 4096 00000080:00200000:2.0:1789760749.473987:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.473994:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.473996:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25370624, 25374720) 0 0 00000080:00200000:2.0:1789760749.474000:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.474012:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6194, 6194] 00020000:00200000:2.0:1789760749.474019:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25370624 chunk: [25370624, 25374720) 25374720, 4096 00020000:00200000:2.0:1789760749.474023:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.474049:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6194, 6194] 00000008:00010000:2.0:1789760749.474064:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.474069:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760749.474078:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25370624, 25374720) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.474089:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25370624=0x1832000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.474132:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.474135:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25374720 00000020:00010000:2.0:1789760749.474141:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.474145:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.474148:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.474151:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.474154:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.474160:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.474165:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.474180:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760749.474259:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 25370624<>25374720 00000100:00100000:2.0:1789760749.474296:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760749.474309:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512654208:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760749.507274:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973513 gen 3 00000008:00010000:3.0:1789760749.507343:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760749.507353:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.507375:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.507406:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760749.507438:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512654208:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760749.507596:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6194, 6194] 00000080:00200000:2.0:1789760749.507612:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.507618:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.507635:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25374720 i_blocks 51207 00000020:00200000:2.0:1789760749.507639:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.507649:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.507685:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25374720, count: 4096 00000080:00200000:2.0:1789760749.507693:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25374720, bytes: 4096 00000080:00200000:2.0:1789760749.507702:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.507711:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.507713:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25374720, 25378816) 0 0 00000080:00200000:2.0:1789760749.507717:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.507747:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6195, 6195] 00020000:00200000:2.0:1789760749.507757:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25374720 chunk: [25374720, 25378816) 25378816, 4096 00020000:00200000:2.0:1789760749.507761:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.507782:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6195, 6195] 00000008:00010000:2.0:1789760749.507801:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.507807:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760749.507818:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25374720, 25378816) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.507831:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25374720=0x1833000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.507891:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.507894:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25378816 00000020:00010000:2.0:1789760749.508068:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.508073:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.508086:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.508090:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.508092:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.508104:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.508112:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.508131:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760749.508242:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 25374720<>25378816 00000100:00100000:3.0:1789760749.508571:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512654336:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760749.551659:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973514 gen 3 00000008:00010000:3.0:1789760749.551695:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760749.551701:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.551709:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.551734:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760749.551760:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512654336:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760749.552738:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6195, 6195] 00000080:00200000:2.0:1789760749.552765:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.552769:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.552782:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25378816 i_blocks 51207 00000020:00200000:2.0:1789760749.552786:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.552940:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.552968:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25378816, count: 4096 00000080:00200000:2.0:1789760749.552974:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25378816, bytes: 4096 00000080:00200000:2.0:1789760749.552985:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.552993:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.552995:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25378816, 25382912) 0 0 00000080:00200000:2.0:1789760749.552999:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.553011:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6196, 6196] 00020000:00200000:2.0:1789760749.553018:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25378816 chunk: [25378816, 25382912) 25382912, 4096 00020000:00200000:2.0:1789760749.553022:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.553103:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6196, 6196] 00000008:00010000:2.0:1789760749.553119:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.553125:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760749.553133:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25378816, 25382912) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.553145:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25378816=0x1834000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.553193:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.553196:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25382912 00000020:00010000:2.0:1789760749.553203:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.553205:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.553208:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.553211:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.553214:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.553221:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.553226:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.553241:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760749.553320:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 25378816<>25382912 00000100:00100000:3.0:1789760749.553400:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512654464:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760749.586520:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973515 gen 3 00000008:00010000:3.0:1789760749.586556:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760749.586562:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.586570:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.586589:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760749.586610:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512654464:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760749.586700:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6196, 6196] 00000080:00200000:2.0:1789760749.586714:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.586718:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.586730:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25382912 i_blocks 51207 00000020:00200000:2.0:1789760749.586734:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.586742:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.586780:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25382912, count: 4096 00000080:00200000:2.0:1789760749.586796:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25382912, bytes: 4096 00000080:00200000:2.0:1789760749.586805:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.586812:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.586814:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25382912, 25387008) 0 0 00000080:00200000:2.0:1789760749.586818:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.586832:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6197, 6197] 00020000:00200000:2.0:1789760749.586839:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25382912 chunk: [25382912, 25387008) 25387008, 4096 00020000:00200000:2.0:1789760749.586843:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.586858:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6197, 6197] 00000008:00010000:2.0:1789760749.586871:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.586875:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760749.586884:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25382912, 25387008) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.586894:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25382912=0x1835000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.586964:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.586966:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25387008 00000020:00010000:2.0:1789760749.586973:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.586976:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.586979:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.586983:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.586986:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.586992:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.586999:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.587014:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760749.587102:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 25382912<>25387008 00000100:00100000:3.0:1789760749.587294:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512654592:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760749.633656:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760749.633667:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.633675:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.633691:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760749.633713:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512654592:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760749.634184:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6197, 6197] 00000080:00200000:2.0:1789760749.634197:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.634202:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.634213:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25387008 i_blocks 51207 00000020:00200000:2.0:1789760749.634216:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.634223:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.634246:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25387008, count: 4096 00000080:00200000:2.0:1789760749.634252:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25387008, bytes: 4096 00000080:00200000:2.0:1789760749.634259:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.634265:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.634267:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25387008, 25391104) 0 0 00000080:00200000:2.0:1789760749.634270:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.634282:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6198, 6198] 00020000:00200000:2.0:1789760749.634288:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25387008 chunk: [25387008, 25391104) 25391104, 4096 00020000:00200000:2.0:1789760749.634291:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.634305:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6198, 6198] 00000008:00010000:2.0:1789760749.634317:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.634321:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760749.634330:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25387008, 25391104) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.634340:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25387008=0x1836000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.634391:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.634394:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25391104 00000020:00010000:2.0:1789760749.634401:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.634404:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.634407:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.634410:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.634413:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.634430:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.634436:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.634449:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760749.634521:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 25387008<>25391104 00000100:00100000:3.0:1789760749.634655:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760749.634668:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512654720:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760749.675588:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973516 gen 3 00000100:00100000:3.0:1789760749.675616:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368f480 x1876699512654720/t4294973517(4294973517) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760765 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760749.675650:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760749.675656:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.675663:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.675680:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760749.675699:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512654720:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760749.676306:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6198, 6198] 00000080:00200000:2.0:1789760749.676320:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.676325:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.676336:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25391104 i_blocks 51207 00000020:00200000:2.0:1789760749.676339:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.676347:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.676379:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25391104, count: 4096 00000080:00200000:2.0:1789760749.676385:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25391104, bytes: 4096 00000080:00200000:2.0:1789760749.676393:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.676400:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.676402:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25391104, 25395200) 0 0 00000080:00200000:2.0:1789760749.676406:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.676428:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6199, 6199] 00020000:00200000:2.0:1789760749.676435:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25391104 chunk: [25391104, 25395200) 25395200, 4096 00020000:00200000:2.0:1789760749.676438:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.676453:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6199, 6199] 00000008:00010000:2.0:1789760749.676465:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.676469:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760749.676478:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25391104, 25395200) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.676488:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25391104=0x1837000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.676529:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.676531:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25395200 00000020:00010000:2.0:1789760749.676537:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.676540:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.676543:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.676546:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.676549:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.676555:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.676560:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.676573:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760749.676643:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 25391104<>25395200 00000100:00100000:3.0:1789760749.676735:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512654848:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760749.714615:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973518 gen 3 00000008:00010000:3.0:1789760749.714660:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760749.714666:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.714673:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.714691:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760749.714715:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512654848:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760749.715219:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6199, 6199] 00000080:00200000:2.0:1789760749.715232:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.715237:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.715252:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25395200 i_blocks 51207 00000020:00200000:2.0:1789760749.715255:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.715263:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.715286:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25395200, count: 4096 00000080:00200000:2.0:1789760749.715291:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25395200, bytes: 4096 00000080:00200000:2.0:1789760749.715299:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.715307:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.715309:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25395200, 25399296) 0 0 00000080:00200000:2.0:1789760749.715312:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.715325:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6200, 6200] 00020000:00200000:2.0:1789760749.715331:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25395200 chunk: [25395200, 25399296) 25399296, 4096 00020000:00200000:2.0:1789760749.715335:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.715350:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6200, 6200] 00000008:00010000:2.0:1789760749.715374:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.715378:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760749.715387:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25395200, 25399296) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.715397:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25395200=0x1838000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.715440:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.715443:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25399296 00000020:00010000:2.0:1789760749.715449:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.715452:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.715455:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.715457:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.715460:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.715571:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.715582:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.715596:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760749.715669:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 25395200<>25399296 00000100:00100000:3.0:1789760749.715773:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760749.715800:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512654976:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760749.756437:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973519 gen 3 00000008:00010000:3.0:1789760749.756483:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760749.756489:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.756498:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.756517:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760749.756551:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512654976:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760749.757420:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6200, 6200] 00000080:00200000:2.0:1789760749.757434:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.757440:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.757463:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25399296 i_blocks 51207 00000020:00200000:2.0:1789760749.757467:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.757475:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.757502:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25399296, count: 4096 00000080:00200000:2.0:1789760749.757509:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25399296, bytes: 4096 00000080:00200000:2.0:1789760749.757520:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.757538:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.757541:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25399296, 25403392) 0 0 00000080:00200000:2.0:1789760749.757545:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.757560:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6201, 6201] 00020000:00200000:2.0:1789760749.757566:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25399296 chunk: [25399296, 25403392) 25403392, 4096 00020000:00200000:2.0:1789760749.757570:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.757588:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6201, 6201] 00000008:00010000:2.0:1789760749.757603:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.757608:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760749.757617:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25399296, 25403392) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.757628:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25399296=0x1839000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.757677:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.757679:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25403392 00000020:00010000:2.0:1789760749.757686:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.757689:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.757693:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.757696:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.757699:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.757706:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.757713:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.757729:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760749.757821:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 25399296<>25403392 00000100:00100000:3.0:1789760749.758287:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512655104:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760749.819332:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973520 gen 3 00000008:00010000:3.0:1789760749.819376:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760749.819383:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.819392:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.819409:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760749.819432:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512655104:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760749.819529:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6201, 6201] 00000080:00200000:2.0:1789760749.819544:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.819549:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.819561:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25403392 i_blocks 51207 00000020:00200000:2.0:1789760749.819564:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.819571:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.819597:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25403392, count: 4096 00000080:00200000:2.0:1789760749.819603:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25403392, bytes: 4096 00000080:00200000:2.0:1789760749.819610:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.819617:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.819619:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25403392, 25407488) 0 0 00000080:00200000:2.0:1789760749.819622:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.819636:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6202, 6202] 00020000:00200000:2.0:1789760749.819642:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25403392 chunk: [25403392, 25407488) 25407488, 4096 00020000:00200000:2.0:1789760749.819646:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.819660:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6202, 6202] 00000008:00010000:2.0:1789760749.819673:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.819678:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760749.819688:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25403392, 25407488) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.819699:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25403392=0x183a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.819733:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.819736:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25407488 00000020:00010000:2.0:1789760749.819742:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.819745:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.819748:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.819761:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.819764:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.819769:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.819774:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760749.819842:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:3.0:1789760749.820011:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad500 - object 0x240000400:5 offset 25403392<>25407488 00000100:00100000:2.0:1789760749.820287:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760749.820526:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512655232:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760749.870590:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760749.870602:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.870612:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.870633:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760749.870659:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512655232:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760749.870791:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6202, 6202] 00000080:00200000:2.0:1789760749.870802:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.870808:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.870818:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25407488 i_blocks 51207 00000020:00200000:2.0:1789760749.870821:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.870829:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.870864:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25407488, count: 4096 00000080:00200000:2.0:1789760749.870870:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25407488, bytes: 4096 00000080:00200000:2.0:1789760749.870878:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.870885:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.870887:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25407488, 25411584) 0 0 00000080:00200000:2.0:1789760749.870892:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.870929:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6203, 6203] 00020000:00200000:2.0:1789760749.870936:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25407488 chunk: [25407488, 25411584) 25411584, 4096 00020000:00200000:2.0:1789760749.870939:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.870954:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6203, 6203] 00000008:00010000:2.0:1789760749.870968:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.870972:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760749.870980:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25407488, 25411584) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.870990:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25407488=0x183b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.871044:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.871047:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25411584 00000020:00010000:2.0:1789760749.871053:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.871056:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.871058:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.871061:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.871063:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.871070:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.871075:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.871087:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760749.871164:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 25407488<>25411584 00000100:00100000:2.0:1789760749.871201:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760749.871215:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512655360:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760749.924096:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973521 gen 3 00000100:00100000:3.0:1789760749.924129:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368f100 x1876699512655360/t4294973522(4294973522) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760765 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760749.924170:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760749.924175:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.924183:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.924204:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760749.924228:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512655360:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760749.925023:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6203, 6203] 00000080:00200000:2.0:1789760749.925038:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.925043:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.925055:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25411584 i_blocks 51207 00000020:00200000:2.0:1789760749.925058:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.925068:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.925095:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25411584, count: 4096 00000080:00200000:2.0:1789760749.925101:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25411584, bytes: 4096 00000080:00200000:2.0:1789760749.925112:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.925120:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.925122:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25411584, 25415680) 0 0 00000080:00200000:2.0:1789760749.925126:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.925138:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6204, 6204] 00020000:00200000:2.0:1789760749.926183:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25411584 chunk: [25411584, 25415680) 25415680, 4096 00020000:00200000:2.0:1789760749.926191:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.926217:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6204, 6204] 00000008:00010000:2.0:1789760749.926234:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.926239:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760749.926248:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25411584, 25415680) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.926259:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25411584=0x183c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.926309:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.926311:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25415680 00000020:00010000:2.0:1789760749.926318:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.926320:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.926323:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.926326:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.926329:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.926337:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.926342:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.926373:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760749.926455:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 25411584<>25415680 00000100:00100000:3.0:1789760749.930005:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760749.930035:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512655488:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760749.991424:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973523 gen 3 00000008:00010000:3.0:1789760749.991478:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760749.991485:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760749.991493:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760749.991511:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760749.991532:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512655488:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760749.993281:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6204, 6204] 00000080:00200000:2.0:1789760749.993298:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760749.993304:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760749.993317:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25415680 i_blocks 51207 00000020:00200000:2.0:1789760749.993322:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760749.993332:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760749.993356:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25415680, count: 4096 00000080:00200000:2.0:1789760749.993371:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25415680, bytes: 4096 00000080:00200000:2.0:1789760749.993382:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760749.993390:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760749.993392:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25415680, 25419776) 0 0 00000080:00200000:2.0:1789760749.993395:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760749.993408:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6205, 6205] 00020000:00200000:2.0:1789760749.993415:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25415680 chunk: [25415680, 25419776) 25419776, 4096 00020000:00200000:2.0:1789760749.993418:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760749.993434:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6205, 6205] 00000008:00010000:2.0:1789760749.993447:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760749.993451:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760749.993461:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25415680, 25419776) append 0 append lockpos 0 00000080:00200000:2.0:1789760749.993471:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25415680=0x183d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760749.993518:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760749.993520:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25419776 00000020:00010000:2.0:1789760749.993527:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760749.993530:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.993532:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760749.993535:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760749.993538:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760749.993544:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760749.993549:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760749.993563:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760749.993641:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 25415680<>25419776 00000100:00100000:3.0:1789760749.993858:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512655616:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760750.061744:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760750.061763:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760750.061771:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760750.061790:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760750.061815:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512655616:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760750.062136:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6205, 6205] 00000080:00200000:2.0:1789760750.062148:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760750.062153:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760750.062163:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25419776 i_blocks 51207 00000020:00200000:2.0:1789760750.062166:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760750.062174:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760750.062198:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25419776, count: 4096 00000080:00200000:2.0:1789760750.062204:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25419776, bytes: 4096 00000080:00200000:2.0:1789760750.062214:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760750.062222:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760750.062224:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25419776, 25423872) 0 0 00000080:00200000:2.0:1789760750.062227:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760750.062239:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6206, 6206] 00020000:00200000:2.0:1789760750.062246:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25419776 chunk: [25419776, 25423872) 25423872, 4096 00020000:00200000:2.0:1789760750.062249:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760750.062263:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6206, 6206] 00000008:00010000:2.0:1789760750.062275:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760750.062279:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760750.062287:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25419776, 25423872) append 0 append lockpos 0 00000080:00200000:2.0:1789760750.062297:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25419776=0x183e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760750.062337:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760750.062339:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25423872 00000020:00010000:2.0:1789760750.062345:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760750.062348:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.062350:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.062353:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760750.062355:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760750.062372:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760750.062378:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760750.062391:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760750.062460:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 25419776<>25423872 00000100:00100000:3.0:1789760750.062601:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760750.062615:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512655744:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760750.110347:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973525 gen 3 00000008:00010000:3.0:1789760750.110597:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760750.110604:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760750.110613:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760750.110634:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760750.110658:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512655744:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760750.110746:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6206, 6206] 00000080:00200000:2.0:1789760750.110822:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760750.110828:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760750.110839:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25423872 i_blocks 51207 00000020:00200000:2.0:1789760750.110842:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760750.110851:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760750.111040:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25423872, count: 4096 00000080:00200000:2.0:1789760750.111048:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25423872, bytes: 4096 00000080:00200000:2.0:1789760750.111056:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760750.111063:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760750.111065:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25423872, 25427968) 0 0 00000080:00200000:2.0:1789760750.111069:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760750.111081:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6207, 6207] 00020000:00200000:2.0:1789760750.111088:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25423872 chunk: [25423872, 25427968) 25427968, 4096 00020000:00200000:2.0:1789760750.111092:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760750.111108:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6207, 6207] 00000008:00010000:2.0:1789760750.111122:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760750.111127:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760750.111136:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25423872, 25427968) append 0 append lockpos 0 00000080:00200000:2.0:1789760750.111146:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25423872=0x183f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760750.111189:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760750.111192:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25427968 00000020:00010000:2.0:1789760750.111199:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760750.111202:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.111205:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.111209:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760750.111212:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760750.111218:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760750.111224:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760750.111238:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760750.111317:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 25423872<>25427968 00000100:00100000:3.0:1789760750.111554:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512655872:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760750.154653:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973526 gen 3 00000008:00010000:3.0:1789760750.154689:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760750.154694:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760750.154703:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760750.154720:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760750.154743:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512655872:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760750.154931:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6207, 6207] 00000080:00200000:2.0:1789760750.154947:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760750.154953:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760750.154965:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25427968 i_blocks 51207 00000020:00200000:2.0:1789760750.154968:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760750.154983:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760750.155017:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25427968, count: 4096 00000080:00200000:2.0:1789760750.155023:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25427968, bytes: 4096 00000080:00200000:2.0:1789760750.155166:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760750.155173:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760750.155175:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25427968, 25432064) 0 0 00000080:00200000:2.0:1789760750.155178:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760750.155190:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6208, 6208] 00020000:00200000:2.0:1789760750.155196:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25427968 chunk: [25427968, 25432064) 25432064, 4096 00020000:00200000:2.0:1789760750.155200:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760750.155214:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6208, 6208] 00000008:00010000:2.0:1789760750.155226:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760750.155230:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760750.155238:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25427968, 25432064) append 0 append lockpos 0 00000080:00200000:2.0:1789760750.155246:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25427968=0x1840000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760750.155283:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760750.155285:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25432064 00000020:00010000:2.0:1789760750.155291:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760750.155293:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.155296:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.155299:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760750.155301:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760750.155307:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760750.155311:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760750.155323:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760750.155397:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 25427968<>25432064 00000100:00100000:3.0:1789760750.155470:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760750.155483:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512656000:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760750.192502:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973527 gen 3 00000008:00010000:3.0:1789760750.192540:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760750.192546:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760750.192554:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760750.192572:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760750.192597:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512656000:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760750.192688:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6208, 6208] 00000080:00200000:2.0:1789760750.192700:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760750.192706:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760750.192719:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25432064 i_blocks 51207 00000020:00200000:2.0:1789760750.192723:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760750.192732:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760750.192767:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25432064, count: 4096 00000080:00200000:2.0:1789760750.192774:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25432064, bytes: 4096 00000080:00200000:2.0:1789760750.192782:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760750.192800:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760750.192802:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25432064, 25436160) 0 0 00000080:00200000:2.0:1789760750.192806:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760750.192818:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6209, 6209] 00020000:00200000:2.0:1789760750.192825:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25432064 chunk: [25432064, 25436160) 25436160, 4096 00020000:00200000:2.0:1789760750.192829:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760750.192847:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6209, 6209] 00000008:00010000:2.0:1789760750.192862:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760750.192867:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760750.192876:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25432064, 25436160) append 0 append lockpos 0 00000080:00200000:2.0:1789760750.192887:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25432064=0x1841000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760750.192956:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760750.192959:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25436160 00000020:00010000:2.0:1789760750.192966:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760750.192969:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.192971:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.192974:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760750.192977:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760750.192983:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760750.193236:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760750.193252:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760750.193333:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 25432064<>25436160 00000100:00100000:3.0:1789760750.193543:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512656128:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760750.227476:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760750.227488:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760750.227496:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760750.227513:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760750.227536:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512656128:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760750.227616:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6209, 6209] 00000080:00200000:2.0:1789760750.227628:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760750.227633:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760750.227644:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25436160 i_blocks 51207 00000020:00200000:2.0:1789760750.227647:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760750.227656:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760750.227680:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25436160, count: 4096 00000080:00200000:2.0:1789760750.227686:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25436160, bytes: 4096 00000080:00200000:2.0:1789760750.227694:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760750.227702:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760750.227704:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25436160, 25440256) 0 0 00000080:00200000:2.0:1789760750.227708:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760750.227720:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6210, 6210] 00020000:00200000:2.0:1789760750.227727:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25436160 chunk: [25436160, 25440256) 25440256, 4096 00020000:00200000:2.0:1789760750.227730:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760750.227744:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6210, 6210] 00000008:00010000:2.0:1789760750.227866:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760750.227871:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760750.227880:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25436160, 25440256) append 0 append lockpos 0 00000080:00200000:2.0:1789760750.227891:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25436160=0x1842000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760750.227955:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760750.227958:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25440256 00000020:00010000:2.0:1789760750.227964:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760750.227967:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.227970:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.227973:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760750.227976:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760750.227982:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760750.227987:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760750.228000:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760750.228193:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 25436160<>25440256 00000100:00100000:3.0:1789760750.228273:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760750.228286:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512656256:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760750.291171:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973528 gen 3 00000100:00100000:3.0:1789760750.291200:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368c380 x1876699512656256/t4294973529(4294973529) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760766 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760750.291238:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760750.291244:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760750.291251:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760750.291270:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760750.291290:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512656256:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760750.291391:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6210, 6210] 00000080:00200000:2.0:1789760750.291404:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760750.291409:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760750.291421:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25440256 i_blocks 51207 00000020:00200000:2.0:1789760750.291425:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760750.291432:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760750.291458:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25440256, count: 4096 00000080:00200000:2.0:1789760750.291464:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25440256, bytes: 4096 00000080:00200000:2.0:1789760750.291472:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760750.291479:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760750.291481:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25440256, 25444352) 0 0 00000080:00200000:2.0:1789760750.291484:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760750.291497:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6211, 6211] 00020000:00200000:2.0:1789760750.291503:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25440256 chunk: [25440256, 25444352) 25444352, 4096 00020000:00200000:2.0:1789760750.291507:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760750.291522:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6211, 6211] 00000008:00010000:2.0:1789760750.291539:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760750.291544:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760750.291554:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25440256, 25444352) append 0 append lockpos 0 00000080:00200000:2.0:1789760750.291566:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25440256=0x1843000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760750.291614:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760750.291616:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25444352 00000020:00010000:2.0:1789760750.291625:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760750.291628:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.291631:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.291634:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760750.291637:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760750.291644:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760750.291651:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760750.291665:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000008:00100000:2.0:1789760750.291747:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 25440256<>25444352 00000100:00100000:3.0:1789760750.291804:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512656384:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760750.356607:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973529 gen 3 00000100:00100000:3.0:1789760750.356635:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368ca80 x1876699512656384/t4294973530(4294973530) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760766 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760750.356672:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760750.356677:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760750.356684:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760750.356703:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760750.356723:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512656384:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760750.356803:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6211, 6211] 00000080:00200000:2.0:1789760750.356817:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760750.356821:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760750.356832:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25444352 i_blocks 51207 00000020:00200000:2.0:1789760750.356835:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760750.356843:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760750.356868:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25444352, count: 4096 00000080:00200000:2.0:1789760750.356874:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25444352, bytes: 4096 00000080:00200000:2.0:1789760750.356882:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760750.356889:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760750.356891:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25444352, 25448448) 0 0 00000080:00200000:2.0:1789760750.356895:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760750.356932:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6212, 6212] 00020000:00200000:2.0:1789760750.356938:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25444352 chunk: [25444352, 25448448) 25448448, 4096 00020000:00200000:2.0:1789760750.356942:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760750.356956:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6212, 6212] 00000008:00010000:2.0:1789760750.356968:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760750.356973:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760750.356982:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25444352, 25448448) append 0 append lockpos 0 00000080:00200000:2.0:1789760750.356992:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25444352=0x1844000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760750.357040:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760750.357042:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25448448 00000020:00010000:2.0:1789760750.357049:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760750.357052:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.357055:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760750.357058:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760750.357060:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760750.357067:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760750.357072:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760750.357085:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760750.357156:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 25444352<>25448448 00000100:00100000:3.0:1789760750.357212:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512656512:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760750.365169:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760750.399856:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973530 gen 3 00000100:00100000:3.0:1789760750.399886:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368ed80 x1876699512656512/t4294973531(4294973531) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760766 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760750.409171:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:2.0:1789760750.409182:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760750.409190:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760750.409208:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:2.0:1789760750.409233:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512656512:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760750.409308:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6212, 6212] 00000080:00200000:3.0:1789760750.409320:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760750.409325:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760750.409336:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25448448 i_blocks 51207 00000020:00200000:3.0:1789760750.409339:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760750.409347:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760750.409380:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25448448, count: 4096 00000080:00200000:3.0:1789760750.409386:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25448448, bytes: 4096 00000080:00200000:3.0:1789760750.409397:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760750.409404:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760750.409406:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25448448, 25452544) 0 0 00000080:00200000:3.0:1789760750.409410:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760750.409422:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6213, 6213] 00020000:00200000:3.0:1789760750.409428:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25448448 chunk: [25448448, 25452544) 25452544, 4096 00020000:00200000:3.0:1789760750.409432:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760750.409447:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6213, 6213] 00000008:00010000:3.0:1789760750.409460:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760750.409465:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760750.409475:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25448448, 25452544) append 0 append lockpos 0 00000080:00200000:3.0:1789760750.409485:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25448448=0x1845000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760750.409528:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760750.409531:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25452544 00000020:00010000:3.0:1789760750.409537:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760750.409540:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760750.409543:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760750.409546:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760750.409548:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760750.409554:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760750.409560:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760750.409573:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760750.409647:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad500 - object 0x240000400:5 offset 25448448<>25452544 00000100:00100000:2.0:1789760750.409738:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512656640:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760750.491167:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973531 gen 3 00000100:00100000:3.0:1789760750.491200:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc0107ad500 x1876699512656640/t4294973532(4294973532) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760766 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760750.491238:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760750.491243:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760750.491253:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760750.491274:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760750.491294:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512656640:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760750.491399:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6213, 6213] 00000080:00200000:1.0:1789760750.491413:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.491418:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.491430:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25452544 i_blocks 51207 00000020:00200000:1.0:1789760750.491433:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.491442:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.491470:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25452544, count: 4096 00000080:00200000:1.0:1789760750.491477:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25452544, bytes: 4096 00000080:00200000:1.0:1789760750.491484:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.491491:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.491493:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25452544, 25456640) 0 0 00000080:00200000:1.0:1789760750.491497:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.491511:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6214, 6214] 00020000:00200000:1.0:1789760750.491519:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25452544 chunk: [25452544, 25456640) 25456640, 4096 00020000:00200000:1.0:1789760750.491523:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.491538:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6214, 6214] 00000008:00010000:1.0:1789760750.491551:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.491556:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760750.491566:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25452544, 25456640) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.491577:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25452544=0x1846000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.491621:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.491624:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25456640 00000020:00010000:1.0:1789760750.491631:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.491634:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.491637:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.491640:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.491643:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.491649:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.491654:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.491669:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760750.491760:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 25452544<>25456640 00000080:00200000:1.0:1789760750.537283:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6214, 6214] 00000080:00200000:1.0:1789760750.537297:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.537302:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.537314:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25456640 i_blocks 51207 00000020:00200000:1.0:1789760750.537317:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.537325:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.537351:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25456640, count: 4096 00000080:00200000:1.0:1789760750.537367:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25456640, bytes: 4096 00000080:00200000:1.0:1789760750.537375:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.537382:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.537384:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25456640, 25460736) 0 0 00000080:00200000:1.0:1789760750.537388:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.537401:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6215, 6215] 00020000:00200000:1.0:1789760750.537407:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25456640 chunk: [25456640, 25460736) 25460736, 4096 00020000:00200000:1.0:1789760750.537410:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.537424:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6215, 6215] 00000008:00010000:1.0:1789760750.537436:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.537440:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760750.537449:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25456640, 25460736) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.537458:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25456640=0x1847000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.537499:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.537502:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25460736 00000020:00010000:1.0:1789760750.537509:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.537512:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.537514:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.537517:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.537520:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.537526:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.537532:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.537545:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760750.537622:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96680 - object 0x240000400:5 offset 25456640<>25460736 00000080:00200000:1.0:1789760750.617187:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6215, 6215] 00000080:00200000:1.0:1789760750.617199:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.617204:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.617216:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25460736 i_blocks 51207 00000020:00200000:1.0:1789760750.617220:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.617228:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.617257:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25460736, count: 4096 00000080:00200000:1.0:1789760750.617263:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25460736, bytes: 4096 00000080:00200000:1.0:1789760750.617271:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.617278:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.617280:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25460736, 25464832) 0 0 00000080:00200000:1.0:1789760750.617284:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.617296:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6216, 6216] 00020000:00200000:1.0:1789760750.617303:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25460736 chunk: [25460736, 25464832) 25464832, 4096 00020000:00200000:1.0:1789760750.617307:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.617322:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6216, 6216] 00000008:00010000:1.0:1789760750.617336:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.617342:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760750.617351:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25460736, 25464832) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.617378:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25460736=0x1848000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.617431:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.617434:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25464832 00000020:00010000:1.0:1789760750.617440:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.617444:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.617447:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.617450:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.617452:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.617458:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.617464:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.617478:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760750.617552:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97480 - object 0x240000400:5 offset 25460736<>25464832 00000100:00100000:1.0:1789760750.617588:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512657024:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760750.648787:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760750.648799:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760750.648809:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760750.648888:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760750.648940:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512657024:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760750.649401:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6216, 6216] 00000080:00200000:3.0:1789760750.649414:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760750.649419:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760750.649430:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25464832 i_blocks 51207 00000020:00200000:3.0:1789760750.649433:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760750.649442:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760750.649469:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25464832, count: 4096 00000080:00200000:3.0:1789760750.649476:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25464832, bytes: 4096 00000080:00200000:3.0:1789760750.649487:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760750.649494:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760750.649496:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25464832, 25468928) 0 0 00000080:00200000:3.0:1789760750.649500:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760750.649514:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6217, 6217] 00020000:00200000:3.0:1789760750.649520:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25464832 chunk: [25464832, 25468928) 25468928, 4096 00020000:00200000:3.0:1789760750.649602:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760750.649621:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6217, 6217] 00000008:00010000:3.0:1789760750.649635:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760750.649640:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760750.649649:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25464832, 25468928) append 0 append lockpos 0 00000080:00200000:3.0:1789760750.649661:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25464832=0x1849000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760750.649709:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760750.649711:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25468928 00000020:00010000:3.0:1789760750.649717:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760750.649721:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760750.649724:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760750.649728:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760750.649731:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760750.649737:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760750.649743:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760750.649759:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760750.649936:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac700 - object 0x240000400:5 offset 25464832<>25468928 00000100:00100000:3.0:1789760750.649977:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760750.649992:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512657152:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760750.705549:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973536 gen 3 00000008:00010000:3.0:1789760750.705597:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760750.705605:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760750.705614:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760750.705638:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760750.705660:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512657152:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760750.705765:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6217, 6217] 00000080:00200000:1.0:1789760750.705778:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.705783:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.705794:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25468928 i_blocks 51207 00000020:00200000:1.0:1789760750.705798:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.705805:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.705831:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25468928, count: 4096 00000080:00200000:1.0:1789760750.705837:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25468928, bytes: 4096 00000080:00200000:1.0:1789760750.705844:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.705851:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.705853:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25468928, 25473024) 0 0 00000080:00200000:1.0:1789760750.705857:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.705869:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6218, 6218] 00020000:00200000:1.0:1789760750.705876:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25468928 chunk: [25468928, 25473024) 25473024, 4096 00020000:00200000:1.0:1789760750.705879:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.705894:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6218, 6218] 00000008:00010000:1.0:1789760750.705930:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.705934:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760750.705944:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25468928, 25473024) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.705954:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25468928=0x184a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.705998:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.706000:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25473024 00000020:00010000:1.0:1789760750.706007:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.706010:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.706013:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.706016:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.706019:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.706037:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.706042:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.706055:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760750.706130:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95880 - object 0x240000400:5 offset 25468928<>25473024 00000100:00100000:1.0:1789760750.706167:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760750.706179:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512657280:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760750.776946:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973537 gen 3 00000008:00010000:1.0:1789760750.776983:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760750.776990:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760750.776999:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760750.777021:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760750.777056:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512657280:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760750.777185:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6218, 6218] 00000080:00200000:3.0:1789760750.777201:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760750.777206:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760750.777218:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25473024 i_blocks 51207 00000020:00200000:3.0:1789760750.777221:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760750.777230:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760750.777256:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25473024, count: 4096 00000080:00200000:3.0:1789760750.777262:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25473024, bytes: 4096 00000080:00200000:3.0:1789760750.777270:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760750.777277:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760750.777279:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25473024, 25477120) 0 0 00000080:00200000:3.0:1789760750.777283:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760750.777295:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6219, 6219] 00020000:00200000:3.0:1789760750.777301:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25473024 chunk: [25473024, 25477120) 25477120, 4096 00020000:00200000:3.0:1789760750.777305:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760750.777321:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6219, 6219] 00000008:00010000:3.0:1789760750.777336:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760750.777341:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760750.777350:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25473024, 25477120) append 0 append lockpos 0 00000080:00200000:3.0:1789760750.777368:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25473024=0x184b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760750.777414:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760750.777417:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25477120 00000020:00010000:3.0:1789760750.777423:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760750.777427:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760750.777429:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760750.777432:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760750.777435:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760750.777441:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760750.777447:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760750.777461:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760750.777549:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac700 - object 0x240000400:5 offset 25473024<>25477120 00000100:00100000:2.0:1789760750.777717:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512657408:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760750.861329:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973538 gen 3 00000008:00010000:3.0:1789760750.861382:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760750.861388:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760750.861399:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760750.861420:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760750.861443:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512657408:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760750.861462:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6219, 6219] 00000080:00200000:1.0:1789760750.861477:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.861483:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.861495:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25477120 i_blocks 51207 00000020:00200000:1.0:1789760750.861499:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.861507:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.861533:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25477120, count: 4096 00000080:00200000:1.0:1789760750.861539:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25477120, bytes: 4096 00000080:00200000:1.0:1789760750.861546:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.861553:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.861555:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25477120, 25481216) 0 0 00000080:00200000:1.0:1789760750.861559:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.861570:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6220, 6220] 00020000:00200000:1.0:1789760750.861577:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25477120 chunk: [25477120, 25481216) 25481216, 4096 00020000:00200000:1.0:1789760750.861580:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.861605:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6220, 6220] 00000008:00010000:1.0:1789760750.861619:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.861623:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760750.861632:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25477120, 25481216) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.861642:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25477120=0x184c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.861684:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.861686:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25481216 00000020:00010000:1.0:1789760750.861693:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.861696:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.861699:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.861702:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.861705:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.861711:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.861716:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.861729:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760750.861804:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95880 - object 0x240000400:5 offset 25477120<>25481216 00000080:00200000:1.0:1789760750.955147:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6220, 6220] 00000080:00200000:1.0:1789760750.955159:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.955165:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.955177:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25481216 i_blocks 51207 00000020:00200000:1.0:1789760750.955180:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.955187:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.955211:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25481216, count: 4096 00000080:00200000:1.0:1789760750.955217:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25481216, bytes: 4096 00000080:00200000:1.0:1789760750.955224:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.955231:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.955233:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25481216, 25485312) 0 0 00000080:00200000:1.0:1789760750.955237:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.955250:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6221, 6221] 00020000:00200000:1.0:1789760750.955256:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25481216 chunk: [25481216, 25485312) 25485312, 4096 00020000:00200000:1.0:1789760750.955260:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.955275:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6221, 6221] 00000008:00010000:1.0:1789760750.955288:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.955292:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760750.955303:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25481216, 25485312) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.955312:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25481216=0x184d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.955738:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.955742:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25485312 00000020:00010000:1.0:1789760750.955850:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.955854:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.955857:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.955861:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.955863:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.955870:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.955876:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.955895:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760750.956014:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96d80 - object 0x240000400:5 offset 25481216<>25485312 00000080:00200000:1.0:1789760750.992975:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6221, 6221] 00000080:00200000:1.0:1789760750.992988:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760750.992993:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760750.993005:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25485312 i_blocks 51207 00000020:00200000:1.0:1789760750.993008:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760750.993015:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760750.993041:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25485312, count: 4096 00000080:00200000:1.0:1789760750.993047:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25485312, bytes: 4096 00000080:00200000:1.0:1789760750.993055:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760750.993062:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760750.993065:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25485312, 25489408) 0 0 00000080:00200000:1.0:1789760750.993069:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760750.993082:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6222, 6222] 00020000:00200000:1.0:1789760750.993089:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25485312 chunk: [25485312, 25489408) 25489408, 4096 00020000:00200000:1.0:1789760750.993092:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760750.993108:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6222, 6222] 00000008:00010000:1.0:1789760750.993123:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760750.993127:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760750.993136:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25485312, 25489408) append 0 append lockpos 0 00000080:00200000:1.0:1789760750.993147:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25485312=0x184e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760750.993189:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760750.993192:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25489408 00000020:00010000:1.0:1789760750.993199:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760750.993201:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.993204:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760750.993207:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760750.993209:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760750.993215:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760750.993221:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760750.993236:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760750.993310:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96d80 - object 0x240000400:5 offset 25485312<>25489408 00000080:00200000:1.0:1789760751.050708:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6222, 6222] 00000080:00200000:1.0:1789760751.050722:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.050726:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.050738:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25489408 i_blocks 51207 00000020:00200000:1.0:1789760751.050741:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.050749:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.050776:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25489408, count: 4096 00000080:00200000:1.0:1789760751.050782:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25489408, bytes: 4096 00000080:00200000:1.0:1789760751.050792:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.050801:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.050804:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25489408, 25493504) 0 0 00000080:00200000:1.0:1789760751.050808:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.050821:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6223, 6223] 00020000:00200000:1.0:1789760751.050828:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25489408 chunk: [25489408, 25493504) 25493504, 4096 00020000:00200000:1.0:1789760751.050832:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.051057:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6223, 6223] 00000008:00010000:1.0:1789760751.051072:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.051077:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760751.051086:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25489408, 25493504) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.051099:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25489408=0x184f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.051142:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.051144:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25493504 00000020:00010000:1.0:1789760751.051151:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.051154:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.051157:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.051160:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.051163:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.051170:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.051176:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.051190:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760751.051266:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95880 - object 0x240000400:5 offset 25489408<>25493504 00000100:00100000:1.0:1789760751.051306:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760751.051320:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512657920:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760751.120475:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973542 gen 3 00000008:00010000:1.0:1789760751.120513:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760751.120520:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760751.120528:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760751.120550:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760751.120571:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512657920:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760751.120664:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6223, 6223] 00000080:00200000:3.0:1789760751.120676:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760751.120682:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760751.120694:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25493504 i_blocks 51207 00000020:00200000:3.0:1789760751.120697:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760751.120705:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760751.120730:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25493504, count: 4096 00000080:00200000:3.0:1789760751.120737:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25493504, bytes: 4096 00000080:00200000:3.0:1789760751.120754:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760751.120762:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760751.120763:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25493504, 25497600) 0 0 00000080:00200000:3.0:1789760751.120767:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760751.120780:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6224, 6224] 00020000:00200000:3.0:1789760751.120787:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25493504 chunk: [25493504, 25497600) 25497600, 4096 00020000:00200000:3.0:1789760751.120790:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760751.120803:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6224, 6224] 00000008:00010000:3.0:1789760751.120816:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760751.120820:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760751.120829:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25493504, 25497600) append 0 append lockpos 0 00000080:00200000:3.0:1789760751.120840:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25493504=0x1850000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760751.120882:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760751.120885:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25497600 00000020:00010000:3.0:1789760751.120891:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760751.120894:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.120897:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760751.120933:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760751.120944:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760751.120950:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760751.120955:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760751.120967:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760751.121038:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac700 - object 0x240000400:5 offset 25493504<>25497600 00000100:00100000:3.0:1789760751.121069:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760751.121083:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512658048:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760751.173897:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973543 gen 3 00000008:00010000:3.0:1789760751.173958:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760751.173964:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760751.173974:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760751.173991:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760751.174014:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512658048:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760751.174103:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6224, 6224] 00000080:00200000:2.0:1789760751.174117:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760751.174122:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760751.174134:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25497600 i_blocks 51207 00000020:00200000:2.0:1789760751.174138:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760751.174145:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760751.174170:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25497600, count: 4096 00000080:00200000:2.0:1789760751.174177:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25497600, bytes: 4096 00000080:00200000:2.0:1789760751.174186:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760751.174194:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760751.174196:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25497600, 25501696) 0 0 00000080:00200000:2.0:1789760751.174200:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760751.174213:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6225, 6225] 00020000:00200000:2.0:1789760751.174221:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25497600 chunk: [25497600, 25501696) 25501696, 4096 00020000:00200000:2.0:1789760751.174225:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760751.174242:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6225, 6225] 00000008:00010000:2.0:1789760751.174255:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760751.174259:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760751.174269:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25497600, 25501696) append 0 append lockpos 0 00000080:00200000:2.0:1789760751.174279:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25497600=0x1851000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760751.174323:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760751.174325:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25501696 00000020:00010000:2.0:1789760751.174333:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760751.174336:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.174339:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.174342:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760751.174345:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760751.174361:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760751.174367:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760751.174381:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760751.174457:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 25497600<>25501696 00000100:00100000:2.0:1789760751.174494:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760751.174507:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512658176:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760751.216633:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973544 gen 3 00000008:00010000:3.0:1789760751.216679:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760751.216685:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760751.216693:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760751.216710:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760751.216735:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512658176:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760751.216818:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6225, 6225] 00000080:00200000:2.0:1789760751.216832:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760751.216837:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760751.216849:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25501696 i_blocks 51207 00000020:00200000:2.0:1789760751.216852:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760751.216861:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760751.216885:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25501696, count: 4096 00000080:00200000:2.0:1789760751.216891:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25501696, bytes: 4096 00000080:00200000:2.0:1789760751.216921:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760751.216929:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760751.216930:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25501696, 25505792) 0 0 00000080:00200000:2.0:1789760751.216934:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760751.216946:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6226, 6226] 00020000:00200000:2.0:1789760751.216951:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25501696 chunk: [25501696, 25505792) 25505792, 4096 00020000:00200000:2.0:1789760751.216955:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760751.216968:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6226, 6226] 00000008:00010000:2.0:1789760751.216981:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760751.216985:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760751.216994:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25501696, 25505792) append 0 append lockpos 0 00000080:00200000:2.0:1789760751.217005:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25501696=0x1852000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760751.217048:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760751.217050:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25505792 00000020:00010000:2.0:1789760751.217057:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760751.217061:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.217075:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.217079:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760751.217082:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760751.217088:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760751.217094:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760751.217107:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760751.217301:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 25501696<>25505792 00000100:00100000:3.0:1789760751.218102:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760751.218130:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512658304:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760751.259401:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973545 gen 3 00000008:00010000:3.0:1789760751.259447:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760751.259454:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760751.259463:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760751.259480:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760751.259505:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512658304:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760751.259648:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6226, 6226] 00000080:00200000:2.0:1789760751.259662:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760751.259667:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760751.259678:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25505792 i_blocks 51207 00000020:00200000:2.0:1789760751.259682:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760751.259689:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760751.259714:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25505792, count: 4096 00000080:00200000:2.0:1789760751.259721:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25505792, bytes: 4096 00000080:00200000:2.0:1789760751.259728:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760751.259735:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760751.259737:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25505792, 25509888) 0 0 00000080:00200000:2.0:1789760751.259751:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760751.259765:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6227, 6227] 00020000:00200000:2.0:1789760751.259772:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25505792 chunk: [25505792, 25509888) 25509888, 4096 00020000:00200000:2.0:1789760751.259776:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760751.259800:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6227, 6227] 00000008:00010000:2.0:1789760751.259815:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760751.259819:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760751.259828:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25505792, 25509888) append 0 append lockpos 0 00000080:00200000:2.0:1789760751.259981:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25505792=0x1853000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760751.260026:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760751.260029:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25509888 00000020:00010000:2.0:1789760751.260036:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760751.260040:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.260042:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.260046:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760751.260049:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760751.260056:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760751.260062:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760751.260077:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000008:00100000:2.0:1789760751.260221:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 25505792<>25509888 00000100:00100000:3.0:1789760751.260281:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512658432:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760751.296772:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973546 gen 3 00000008:00010000:3.0:1789760751.296805:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760751.296810:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760751.296819:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760751.296836:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760751.296860:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512658432:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760751.297010:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6227, 6227] 00000080:00200000:2.0:1789760751.297022:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760751.297028:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760751.297039:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25509888 i_blocks 51207 00000020:00200000:2.0:1789760751.297042:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760751.297050:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760751.297075:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25509888, count: 4096 00000080:00200000:2.0:1789760751.297081:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25509888, bytes: 4096 00000080:00200000:2.0:1789760751.297088:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760751.297096:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760751.297098:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25509888, 25513984) 0 0 00000080:00200000:2.0:1789760751.297102:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760751.297116:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6228, 6228] 00020000:00200000:2.0:1789760751.297124:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25509888 chunk: [25509888, 25513984) 25513984, 4096 00020000:00200000:2.0:1789760751.297127:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760751.297142:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6228, 6228] 00000008:00010000:2.0:1789760751.297154:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760751.297159:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760751.297168:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25509888, 25513984) append 0 append lockpos 0 00000080:00200000:2.0:1789760751.297178:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25509888=0x1854000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760751.297222:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760751.297224:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25513984 00000020:00010000:2.0:1789760751.297231:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760751.297234:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.297237:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.297240:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760751.297242:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760751.297248:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760751.297253:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760751.297266:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000008:00100000:2.0:1789760751.297340:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 25509888<>25513984 00000100:00100000:3.0:1789760751.297436:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512658560:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760751.360200:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973547 gen 3 00000008:00010000:3.0:1789760751.360238:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760751.360244:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760751.360253:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760751.360270:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760751.360292:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512658560:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760751.360430:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6228, 6228] 00000080:00200000:2.0:1789760751.360441:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760751.360446:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760751.360457:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25513984 i_blocks 51207 00000020:00200000:2.0:1789760751.360461:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760751.360468:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760751.360492:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25513984, count: 4096 00000080:00200000:2.0:1789760751.360497:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25513984, bytes: 4096 00000080:00200000:2.0:1789760751.360504:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760751.360511:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760751.360514:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25513984, 25518080) 0 0 00000080:00200000:2.0:1789760751.360518:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760751.360529:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6229, 6229] 00020000:00200000:2.0:1789760751.360535:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25513984 chunk: [25513984, 25518080) 25518080, 4096 00020000:00200000:2.0:1789760751.360539:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760751.360552:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6229, 6229] 00000008:00010000:2.0:1789760751.360564:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760751.360568:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760751.360576:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25513984, 25518080) append 0 append lockpos 0 00000080:00200000:2.0:1789760751.360584:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25513984=0x1855000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760751.360621:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760751.360624:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25518080 00000020:00010000:2.0:1789760751.360630:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760751.360633:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.360635:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.360638:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760751.360641:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760751.360647:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760751.360652:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760751.360663:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760751.360734:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 25513984<>25518080 00000100:00100000:3.0:1789760751.360854:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760751.360867:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512658688:192.168.203.156@tcp:4:dd.0 00010000:00010000:3.0:1789760751.389109:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760751.406390:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973548 gen 3 00000008:00010000:3.0:1789760751.406429:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760751.406436:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760751.406444:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760751.406464:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760751.406486:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512658688:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760751.406574:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6229, 6229] 00000080:00200000:2.0:1789760751.406587:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760751.406591:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760751.406603:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25518080 i_blocks 51207 00000020:00200000:2.0:1789760751.406607:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760751.406614:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760751.406638:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25518080, count: 4096 00000080:00200000:2.0:1789760751.406644:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25518080, bytes: 4096 00000080:00200000:2.0:1789760751.406651:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760751.406657:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760751.406658:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25518080, 25522176) 0 0 00000080:00200000:2.0:1789760751.406662:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760751.406674:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6230, 6230] 00020000:00200000:2.0:1789760751.406680:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25518080 chunk: [25518080, 25522176) 25522176, 4096 00020000:00200000:2.0:1789760751.406684:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760751.406699:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6230, 6230] 00000008:00010000:2.0:1789760751.406712:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760751.406717:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760751.406725:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25518080, 25522176) append 0 append lockpos 0 00000080:00200000:2.0:1789760751.406736:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25518080=0x1856000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760751.406803:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760751.406805:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25522176 00000020:00010000:2.0:1789760751.406813:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760751.406816:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.406819:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760751.406822:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760751.406825:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760751.406831:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760751.406837:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760751.406850:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760751.407291:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 25518080<>25522176 00000100:00100000:3.0:1789760751.407342:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512658816:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760751.444851:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973549 gen 3 00000008:00010000:3.0:1789760751.444888:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760751.444895:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760751.444937:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760751.444960:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760751.444986:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512658816:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760751.445023:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6230, 6230] 00000080:00200000:1.0:1789760751.445036:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.445042:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.445053:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25522176 i_blocks 51207 00000020:00200000:1.0:1789760751.445056:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.445064:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.445088:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25522176, count: 4096 00000080:00200000:1.0:1789760751.445094:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25522176, bytes: 4096 00000080:00200000:1.0:1789760751.445101:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.445108:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.445110:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25522176, 25526272) 0 0 00000080:00200000:1.0:1789760751.445113:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.445125:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6231, 6231] 00020000:00200000:1.0:1789760751.445132:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25522176 chunk: [25522176, 25526272) 25526272, 4096 00020000:00200000:1.0:1789760751.445136:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.445149:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6231, 6231] 00000008:00010000:1.0:1789760751.445161:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.445165:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760751.445174:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25522176, 25526272) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.445183:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25522176=0x1857000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.445223:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.445225:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25526272 00000020:00010000:1.0:1789760751.445231:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.445235:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.445237:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.445241:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.445243:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.445248:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.445253:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.445265:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760751.445409:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95880 - object 0x240000400:5 offset 25522176<>25526272 00000080:00200000:1.0:1789760751.497002:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6231, 6231] 00000080:00200000:1.0:1789760751.497014:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.497082:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.497095:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25526272 i_blocks 51207 00000020:00200000:1.0:1789760751.497099:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.497107:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.497133:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25526272, count: 4096 00000080:00200000:1.0:1789760751.497140:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25526272, bytes: 4096 00000080:00200000:1.0:1789760751.497147:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.497155:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.497157:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25526272, 25530368) 0 0 00000080:00200000:1.0:1789760751.497161:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.497174:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6232, 6232] 00020000:00200000:1.0:1789760751.497180:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25526272 chunk: [25526272, 25530368) 25530368, 4096 00020000:00200000:1.0:1789760751.497184:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.497198:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6232, 6232] 00000008:00010000:1.0:1789760751.497212:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.497216:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760751.497224:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25526272, 25530368) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.497233:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25526272=0x1858000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.497274:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.497276:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25530368 00000020:00010000:1.0:1789760751.497283:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.497286:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.497289:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.497292:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.497294:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.497300:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.497305:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.497318:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760751.497400:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97480 - object 0x240000400:5 offset 25526272<>25530368 00000080:00200000:1.0:1789760751.539415:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6232, 6232] 00000080:00200000:1.0:1789760751.539427:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.539432:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.539444:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25530368 i_blocks 51207 00000020:00200000:1.0:1789760751.539448:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.539456:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.539483:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25530368, count: 4096 00000080:00200000:1.0:1789760751.539490:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25530368, bytes: 4096 00000080:00200000:1.0:1789760751.539496:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.539504:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.539506:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25530368, 25534464) 0 0 00000080:00200000:1.0:1789760751.539510:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.539524:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6233, 6233] 00020000:00200000:1.0:1789760751.539531:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25530368 chunk: [25530368, 25534464) 25534464, 4096 00020000:00200000:1.0:1789760751.539534:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.539551:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6233, 6233] 00000008:00010000:1.0:1789760751.539566:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.539570:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760751.539578:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25530368, 25534464) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.539589:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25530368=0x1859000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.539629:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.539631:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25534464 00000020:00010000:1.0:1789760751.539638:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.539640:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.539643:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.539646:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.539649:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.539654:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.539659:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.539672:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760751.539866:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96680 - object 0x240000400:5 offset 25530368<>25534464 00000080:00200000:1.0:1789760751.577857:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6233, 6233] 00000080:00200000:1.0:1789760751.577868:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.577874:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.577886:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25534464 i_blocks 51207 00000020:00200000:1.0:1789760751.577889:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.577897:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.577939:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25534464, count: 4096 00000080:00200000:1.0:1789760751.577945:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25534464, bytes: 4096 00000080:00200000:1.0:1789760751.577954:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.578133:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.578136:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25534464, 25538560) 0 0 00000080:00200000:1.0:1789760751.578140:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.578153:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6234, 6234] 00020000:00200000:1.0:1789760751.578161:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25534464 chunk: [25534464, 25538560) 25538560, 4096 00020000:00200000:1.0:1789760751.578165:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.578179:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6234, 6234] 00000008:00010000:1.0:1789760751.578193:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.578197:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760751.578206:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25534464, 25538560) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.578216:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25534464=0x185a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.578261:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.578263:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25538560 00000020:00010000:1.0:1789760751.578270:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.578273:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.578275:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.578278:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.578280:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.578285:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.578291:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.578305:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760751.578389:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 25534464<>25538560 00000080:00200000:1.0:1789760751.617963:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6234, 6234] 00000080:00200000:1.0:1789760751.617975:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.617980:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.617992:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25538560 i_blocks 51207 00000020:00200000:1.0:1789760751.617995:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.618003:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.618027:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25538560, count: 4096 00000080:00200000:1.0:1789760751.618042:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25538560, bytes: 4096 00000080:00200000:1.0:1789760751.618049:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.618057:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.618059:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25538560, 25542656) 0 0 00000080:00200000:1.0:1789760751.618062:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.618075:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6235, 6235] 00020000:00200000:1.0:1789760751.618081:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25538560 chunk: [25538560, 25542656) 25542656, 4096 00020000:00200000:1.0:1789760751.618085:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.618100:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6235, 6235] 00000008:00010000:1.0:1789760751.618113:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.618117:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760751.618125:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25538560, 25542656) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.618135:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25538560=0x185b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.618175:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.618178:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25542656 00000020:00010000:1.0:1789760751.618183:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.618187:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.618189:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.618193:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.618195:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.618201:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.618207:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.618219:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760751.618293:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 25538560<>25542656 00000080:00200000:1.0:1789760751.680944:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6235, 6235] 00000080:00200000:1.0:1789760751.680961:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.680965:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.680977:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25542656 i_blocks 51207 00000020:00200000:1.0:1789760751.680981:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.680990:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.681015:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25542656, count: 4096 00000080:00200000:1.0:1789760751.681033:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25542656, bytes: 4096 00000080:00200000:1.0:1789760751.681044:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.681053:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.681055:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25542656, 25546752) 0 0 00000080:00200000:1.0:1789760751.681058:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.681071:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6236, 6236] 00020000:00200000:1.0:1789760751.681078:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25542656 chunk: [25542656, 25546752) 25546752, 4096 00020000:00200000:1.0:1789760751.681081:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.681095:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6236, 6236] 00000008:00010000:1.0:1789760751.681108:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.681114:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760751.681122:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25542656, 25546752) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.681133:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25542656=0x185c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.681180:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.681182:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25546752 00000020:00010000:1.0:1789760751.681188:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.681191:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.681194:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.681196:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.681199:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.681205:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.681211:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.681226:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760751.681301:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 25542656<>25546752 00000080:00200000:1.0:1789760751.762211:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6236, 6236] 00000080:00200000:1.0:1789760751.762223:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.762227:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.762239:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25546752 i_blocks 51207 00000020:00200000:1.0:1789760751.762243:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.762250:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.762457:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25546752, count: 4096 00000080:00200000:1.0:1789760751.762465:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25546752, bytes: 4096 00000080:00200000:1.0:1789760751.762472:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.762480:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.762482:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25546752, 25550848) 0 0 00000080:00200000:1.0:1789760751.762485:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.762498:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6237, 6237] 00020000:00200000:1.0:1789760751.762504:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25546752 chunk: [25546752, 25550848) 25550848, 4096 00020000:00200000:1.0:1789760751.762509:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.762524:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6237, 6237] 00000008:00010000:1.0:1789760751.762538:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.762543:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760751.762550:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25546752, 25550848) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.762561:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25546752=0x185d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.762603:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.762606:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25550848 00000020:00010000:1.0:1789760751.762612:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.762615:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.762618:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.762621:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.762623:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.762629:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.762634:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.762647:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760751.762719:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94380 - object 0x240000400:5 offset 25546752<>25550848 00000080:00200000:1.0:1789760751.819526:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6237, 6237] 00000080:00200000:1.0:1789760751.819547:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.819552:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.819566:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25550848 i_blocks 51207 00000020:00200000:1.0:1789760751.819570:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.819582:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.819609:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25550848, count: 4096 00000080:00200000:1.0:1789760751.819615:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25550848, bytes: 4096 00000080:00200000:1.0:1789760751.819627:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.819635:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.819637:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25550848, 25554944) 0 0 00000080:00200000:1.0:1789760751.819641:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.819653:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6238, 6238] 00020000:00200000:1.0:1789760751.819661:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25550848 chunk: [25550848, 25554944) 25554944, 4096 00020000:00200000:1.0:1789760751.819665:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.819681:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6238, 6238] 00000008:00010000:1.0:1789760751.819696:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.819701:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760751.819710:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25550848, 25554944) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.819722:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25550848=0x185e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.820583:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.820589:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25554944 00000020:00010000:1.0:1789760751.820597:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.820601:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.820604:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.820607:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.820609:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.820617:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.820624:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.820647:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760751.820731:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 25550848<>25554944 00000080:00200000:1.0:1789760751.907868:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6238, 6238] 00000080:00200000:1.0:1789760751.907880:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760751.907885:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760751.907897:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25554944 i_blocks 51207 00000020:00200000:1.0:1789760751.908302:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760751.908312:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760751.908338:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25554944, count: 4096 00000080:00200000:1.0:1789760751.908357:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25554944, bytes: 4096 00000080:00200000:1.0:1789760751.908367:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760751.908375:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760751.908377:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25554944, 25559040) 0 0 00000080:00200000:1.0:1789760751.908381:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760751.908394:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6239, 6239] 00020000:00200000:1.0:1789760751.908401:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25554944 chunk: [25554944, 25559040) 25559040, 4096 00020000:00200000:1.0:1789760751.908405:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760751.908422:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6239, 6239] 00000008:00010000:1.0:1789760751.908436:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760751.908441:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760751.908450:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25554944, 25559040) append 0 append lockpos 0 00000080:00200000:1.0:1789760751.908461:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25554944=0x185f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760751.908503:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760751.908506:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25559040 00000020:00010000:1.0:1789760751.908512:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760751.908514:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.908517:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760751.908520:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760751.908522:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760751.908528:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760751.908534:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760751.908551:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760751.908640:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94700 - object 0x240000400:5 offset 25554944<>25559040 00000100:00100000:1.0:1789760751.908684:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760751.908699:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512659968:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760751.973839:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973558 gen 3 00000008:00010000:1.0:1789760751.973875:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760751.973881:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760751.973889:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760751.973995:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760751.974019:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512659968:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760751.975574:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512660096:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.053269:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973559 gen 3 00000008:00010000:1.0:1789760752.053309:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760752.053317:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.053327:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.053357:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760752.053382:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512660096:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.054053:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760752.054066:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512660224:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.107516:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973560 gen 3 00000008:00010000:1.0:1789760752.107553:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760752.107560:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.107570:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.107637:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760752.107660:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512660224:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.108498:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512660352:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.151886:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973561 gen 3 00000008:00010000:1.0:1789760752.151959:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760752.151966:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.151974:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.152052:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760752.152074:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512660352:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.152975:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760752.152989:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512660480:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.200750:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973562 gen 3 00000008:00010000:1.0:1789760752.200789:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760752.200796:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.200806:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.200830:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760752.200855:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512660480:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.201708:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512660608:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760752.235501:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760752.235513:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.235522:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.235544:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760752.235570:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512660608:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.236374:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760752.236389:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512660736:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.277380:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973563 gen 3 00000100:00100000:1.0:1789760752.277412:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005148a80 x1876699512660736/t4294973564(4294973564) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760768 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760752.277449:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760752.277454:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.277462:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.277481:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760752.277500:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512660736:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.278191:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512660864:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.308404:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973564 gen 3 00000100:00100000:1.0:1789760752.308430:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00514a680 x1876699512660864/t4294973565(4294973565) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760768 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760752.308471:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760752.308476:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.308485:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.308506:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760752.308529:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512660864:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.309220:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760752.309236:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512660992:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.366796:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973565 gen 3 00000100:00100000:1.0:1789760752.366828:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00514a300 x1876699512660992/t4294973566(4294973566) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760768 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760752.366866:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760752.366871:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.366879:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.366933:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760752.366953:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512660992:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.367438:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512661120:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.410850:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973566 gen 3 00000100:00100000:1.0:1789760752.410879:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005149880 x1876699512661120/t4294973567(4294973567) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760768 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760752.410941:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760752.410947:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.410956:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.410983:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760752.411003:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512661120:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.411808:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760752.411825:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512661248:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760752.414190:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760752.438980:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973567 gen 3 00000100:00100000:1.0:1789760752.439085:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00514b480 x1876699512661248/t4294973568(4294973568) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760768 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760752.439123:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760752.439127:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.439134:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.439382:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760752.439403:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512661248:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.439837:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512661376:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.474444:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973569 gen 3 00000008:00010000:1.0:1789760752.474491:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760752.474499:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.474507:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.474517:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760752.474540:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512661376:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.475513:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512661504:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.509197:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973570 gen 3 00000008:00010000:1.0:1789760752.509236:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760752.509247:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.509256:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.509266:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760752.509287:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512661504:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.509885:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512661632:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.555030:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973571 gen 3 00000008:00010000:1.0:1789760752.555071:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760752.555078:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.555088:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.555112:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760752.555137:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512661632:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.555719:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512661760:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.598689:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973572 gen 3 00000008:00010000:1.0:1789760752.598725:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760752.598731:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.598751:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.598769:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760752.598792:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512661760:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760752.598884:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6253, 6253] 00000080:00200000:2.0:1789760752.598897:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.598928:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.598941:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25616384 i_blocks 51207 00000020:00200000:2.0:1789760752.598944:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.598952:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.598980:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25616384, count: 4096 00000080:00200000:2.0:1789760752.598986:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25616384, bytes: 4096 00000080:00200000:2.0:1789760752.598993:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.599000:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.599003:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25616384, 25620480) 0 0 00000080:00200000:2.0:1789760752.599007:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.599021:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6254, 6254] 00020000:00200000:2.0:1789760752.599028:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25616384 chunk: [25616384, 25620480) 25620480, 4096 00020000:00200000:2.0:1789760752.599031:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.599048:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6254, 6254] 00000008:00010000:2.0:1789760752.599062:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.599067:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760752.599076:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25616384, 25620480) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.599086:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25616384=0x186e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.599133:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.599136:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25620480 00000020:00010000:2.0:1789760752.599142:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.599146:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.599148:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.599152:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.599155:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.599161:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.599167:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.599182:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000008:00100000:2.0:1789760752.599256:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 25616384<>25620480 00000100:00100000:3.0:1789760752.599365:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760752.599380:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512661888:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760752.645720:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973573 gen 3 00000008:00010000:3.0:1789760752.645772:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760752.645778:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.645794:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.645810:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760752.645833:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512661888:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760752.645960:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6254, 6254] 00000080:00200000:2.0:1789760752.645972:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.645977:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.645988:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25620480 i_blocks 51207 00000020:00200000:2.0:1789760752.645991:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.646000:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.646035:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25620480, count: 4096 00000080:00200000:2.0:1789760752.646042:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25620480, bytes: 4096 00000080:00200000:2.0:1789760752.646048:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.646055:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.646057:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25620480, 25624576) 0 0 00000080:00200000:2.0:1789760752.646071:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.646084:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6255, 6255] 00020000:00200000:2.0:1789760752.646090:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25620480 chunk: [25620480, 25624576) 25624576, 4096 00020000:00200000:2.0:1789760752.646094:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.646109:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6255, 6255] 00000008:00010000:2.0:1789760752.646122:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.646127:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760752.646136:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25620480, 25624576) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.646146:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25620480=0x186f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.646189:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.646192:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25624576 00000020:00010000:2.0:1789760752.646198:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.646201:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.646203:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.646206:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.646209:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.646215:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.646221:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.646234:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760752.646306:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 25620480<>25624576 00000100:00100000:3.0:1789760752.646396:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760752.646410:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512662016:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760752.693935:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973574 gen 3 00000008:00010000:3.0:1789760752.693969:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760752.693975:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.693983:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.694105:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760752.694132:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512662016:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760752.694204:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6255, 6255] 00000080:00200000:2.0:1789760752.694216:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.694221:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.694232:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25624576 i_blocks 51207 00000020:00200000:2.0:1789760752.694236:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.694244:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.694270:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25624576, count: 4096 00000080:00200000:2.0:1789760752.694277:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25624576, bytes: 4096 00000080:00200000:2.0:1789760752.694284:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.694291:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.694293:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25624576, 25628672) 0 0 00000080:00200000:2.0:1789760752.694297:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.694310:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6256, 6256] 00020000:00200000:2.0:1789760752.694317:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25624576 chunk: [25624576, 25628672) 25628672, 4096 00020000:00200000:2.0:1789760752.694320:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.694334:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6256, 6256] 00000008:00010000:2.0:1789760752.694357:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.694362:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760752.694371:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25624576, 25628672) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.694381:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25624576=0x1870000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.694423:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.694426:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25628672 00000020:00010000:2.0:1789760752.694432:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.694434:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.694437:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.694441:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.694443:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.694449:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.694455:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.694469:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760752.694541:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 25624576<>25628672 00000100:00100000:3.0:1789760752.694598:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512662144:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760752.734846:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973575 gen 3 00000008:00010000:3.0:1789760752.734883:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760752.734889:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.734896:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.734973:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760752.734999:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512662144:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760752.736230:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6256, 6256] 00000080:00200000:2.0:1789760752.736246:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.736251:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.736264:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25628672 i_blocks 51207 00000020:00200000:2.0:1789760752.736268:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.736277:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.736306:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25628672, count: 4096 00000080:00200000:2.0:1789760752.736314:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25628672, bytes: 4096 00000080:00200000:2.0:1789760752.736322:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.736329:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.736331:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25628672, 25632768) 0 0 00000080:00200000:2.0:1789760752.736334:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.736357:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6257, 6257] 00020000:00200000:2.0:1789760752.736364:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25628672 chunk: [25628672, 25632768) 25632768, 4096 00020000:00200000:2.0:1789760752.736368:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.736383:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6257, 6257] 00000008:00010000:2.0:1789760752.736397:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.736402:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760752.736411:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25628672, 25632768) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.736421:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25628672=0x1871000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.736467:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.736470:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25632768 00000020:00010000:2.0:1789760752.736476:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.736479:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.736482:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.736485:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.736488:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.736494:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.736499:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.736518:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760752.736596:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 25628672<>25632768 00000100:00100000:3.0:1789760752.738007:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760752.738029:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512662272:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760752.779200:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760752.779212:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.779220:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.779238:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760752.779262:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512662272:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760752.779352:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6257, 6257] 00000080:00200000:2.0:1789760752.779365:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.779371:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.779383:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25632768 i_blocks 51207 00000020:00200000:2.0:1789760752.779387:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.779394:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.779418:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25632768, count: 4096 00000080:00200000:2.0:1789760752.779424:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25632768, bytes: 4096 00000080:00200000:2.0:1789760752.779431:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.779438:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.779440:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25632768, 25636864) 0 0 00000080:00200000:2.0:1789760752.779443:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.779460:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6258, 6258] 00020000:00200000:2.0:1789760752.779467:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25632768 chunk: [25632768, 25636864) 25636864, 4096 00020000:00200000:2.0:1789760752.779470:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.779485:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6258, 6258] 00000008:00010000:2.0:1789760752.779498:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.779502:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760752.779511:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25632768, 25636864) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.779521:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25632768=0x1872000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.779562:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.779565:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25636864 00000020:00010000:2.0:1789760752.779571:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.779574:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.779576:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.779580:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.779582:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.779588:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.779594:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.779607:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760752.779715:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 25632768<>25636864 00000100:00100000:2.0:1789760752.779833:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760752.779848:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512662400:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760752.823442:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973576 gen 3 00000100:00100000:3.0:1789760752.823476:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368e680 x1876699512662400/t4294973577(4294973577) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760768 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760752.823516:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760752.823523:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.823532:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.823552:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760752.823572:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512662400:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760752.823664:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6258, 6258] 00000080:00200000:2.0:1789760752.823677:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.823682:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.823694:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25636864 i_blocks 51207 00000020:00200000:2.0:1789760752.823698:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.823705:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.823744:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25636864, count: 4096 00000080:00200000:2.0:1789760752.823750:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25636864, bytes: 4096 00000080:00200000:2.0:1789760752.823758:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.823765:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.823767:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25636864, 25640960) 0 0 00000080:00200000:2.0:1789760752.823771:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.823784:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6259, 6259] 00020000:00200000:2.0:1789760752.823791:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25636864 chunk: [25636864, 25640960) 25640960, 4096 00020000:00200000:2.0:1789760752.823795:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.823811:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6259, 6259] 00000008:00010000:2.0:1789760752.823825:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.823830:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760752.823839:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25636864, 25640960) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.823850:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25636864=0x1873000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.823891:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.823893:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25640960 00000020:00010000:2.0:1789760752.823938:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.823943:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.823946:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.823949:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.823952:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.823959:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.823966:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.823980:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760752.824063:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 25636864<>25640960 00000100:00100000:3.0:1789760752.824259:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512662528:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760752.881047:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973577 gen 3 00000100:00100000:3.0:1789760752.881080:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368df80 x1876699512662528/t4294973578(4294973578) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760768 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760752.881117:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760752.881123:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.881131:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.881150:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760752.881174:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512662528:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760752.881257:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6259, 6259] 00000080:00200000:2.0:1789760752.881270:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760752.881275:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760752.881287:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25640960 i_blocks 51207 00000020:00200000:2.0:1789760752.881291:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760752.881298:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760752.881324:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25640960, count: 4096 00000080:00200000:2.0:1789760752.881331:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25640960, bytes: 4096 00000080:00200000:2.0:1789760752.881347:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760752.881355:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760752.881357:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25640960, 25645056) 0 0 00000080:00200000:2.0:1789760752.881360:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760752.881372:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6260, 6260] 00020000:00200000:2.0:1789760752.881379:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25640960 chunk: [25640960, 25645056) 25645056, 4096 00020000:00200000:2.0:1789760752.881382:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760752.881397:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6260, 6260] 00000008:00010000:2.0:1789760752.881411:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760752.881416:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760752.881425:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25640960, 25645056) append 0 append lockpos 0 00000080:00200000:2.0:1789760752.881435:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25640960=0x1874000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760752.881482:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760752.881485:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25645056 00000020:00010000:2.0:1789760752.881491:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760752.881496:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.881499:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760752.881502:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760752.881504:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760752.881510:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760752.881517:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760752.881530:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760752.881612:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 25640960<>25645056 00000100:00100000:3.0:1789760752.881708:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512662656:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760752.914495:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973578 gen 3 00000100:00100000:3.0:1789760752.914934:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368e300 x1876699512662656/t4294973579(4294973579) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760768 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760752.914976:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760752.914981:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760752.914988:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760752.915008:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760752.915027:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512662656:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.915840:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512662784:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760752.925513:0:38:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1789760752.925531:0:38:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760752.925556:0:38:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:1.0:1789760752.959412:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973580 gen 3 00000008:00010000:1.0:1789760752.959458:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760752.959466:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760752.959475:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760752.959504:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760752.959529:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512662784:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760752.960564:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760752.960581:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512662912:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760753.021456:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760753.021467:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760753.021476:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760753.021496:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760753.021518:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512662912:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760753.023197:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512663040:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760753.058651:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973582 gen 3 00000008:00010000:1.0:1789760753.058697:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760753.058703:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760753.058711:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760753.058796:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760753.058822:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512663040:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760753.059856:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512663168:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760753.097407:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973583 gen 3 00000008:00010000:1.0:1789760753.097443:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760753.097450:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760753.097460:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760753.097526:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760753.097548:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512663168:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760753.098272:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512663296:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760753.125130:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760753.125142:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760753.125151:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760753.125173:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760753.125196:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512663296:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760753.125998:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512663424:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760753.158840:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973584 gen 3 00000100:00100000:1.0:1789760753.158869:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00514bb80 x1876699512663424/t4294973585(4294973585) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760769 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760753.158941:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760753.158948:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760753.158957:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760753.158979:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760753.158998:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512663424:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760753.159527:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760753.159541:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512663552:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760753.199060:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973585 gen 3 00000100:00100000:1.0:1789760753.199089:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005149f80 x1876699512663552/t4294973586(4294973586) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760769 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760753.199125:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760753.199129:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760753.199137:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760753.199152:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760753.199170:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512663552:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760753.199768:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760753.199783:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512663680:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760753.241062:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6268, 6268] 00000080:00200000:2.0:1789760753.241075:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760753.241079:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760753.241090:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25677824 i_blocks 51207 00000020:00200000:2.0:1789760753.241093:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760753.241100:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760753.241127:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25677824, count: 4096 00000080:00200000:2.0:1789760753.241195:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25677824, bytes: 4096 00000080:00200000:2.0:1789760753.241206:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760753.241213:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760753.241215:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25677824, 25681920) 0 0 00000080:00200000:2.0:1789760753.241219:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760753.241231:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6269, 6269] 00020000:00200000:2.0:1789760753.241238:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25677824 chunk: [25677824, 25681920) 25681920, 4096 00020000:00200000:2.0:1789760753.241242:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760753.241258:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6269, 6269] 00000008:00010000:2.0:1789760753.241270:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760753.241274:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760753.241284:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25677824, 25681920) append 0 append lockpos 0 00000080:00200000:2.0:1789760753.241299:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25677824=0x187d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760753.241354:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760753.241357:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25681920 00000020:00010000:2.0:1789760753.241363:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760753.241366:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.241369:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.241373:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760753.241375:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760753.241382:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760753.241388:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760753.241401:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760753.241474:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 25677824<>25681920 00000100:00100000:3.0:1789760753.241704:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512663808:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760753.296965:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973588 gen 3 00000008:00010000:3.0:1789760753.297003:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760753.297010:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760753.297018:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760753.297032:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760753.297054:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512663808:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760753.297205:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6269, 6269] 00000080:00200000:2.0:1789760753.297217:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760753.297222:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760753.297234:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25681920 i_blocks 51207 00000020:00200000:2.0:1789760753.297237:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760753.297245:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760753.297280:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25681920, count: 4096 00000080:00200000:2.0:1789760753.297287:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25681920, bytes: 4096 00000080:00200000:2.0:1789760753.297295:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760753.297302:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760753.297304:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25681920, 25686016) 0 0 00000080:00200000:2.0:1789760753.297307:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760753.297320:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6270, 6270] 00020000:00200000:2.0:1789760753.297327:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25681920 chunk: [25681920, 25686016) 25686016, 4096 00020000:00200000:2.0:1789760753.297331:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760753.297356:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6270, 6270] 00000008:00010000:2.0:1789760753.297377:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760753.297383:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760753.297392:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25681920, 25686016) append 0 append lockpos 0 00000080:00200000:2.0:1789760753.297561:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25681920=0x187e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760753.297605:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760753.297608:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25686016 00000020:00010000:2.0:1789760753.297614:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760753.297619:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.297622:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.297625:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760753.297628:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760753.297635:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760753.297641:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760753.297654:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760753.297738:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 25681920<>25686016 00000100:00100000:3.0:1789760753.298101:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760753.298117:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512663936:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760753.373144:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973589 gen 3 00000008:00010000:3.0:1789760753.373188:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760753.373194:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760753.373203:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760753.373258:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760753.373277:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512663936:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760753.376050:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6270, 6270] 00000080:00200000:2.0:1789760753.376069:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760753.376074:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760753.376087:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25686016 i_blocks 51207 00000020:00200000:2.0:1789760753.376091:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760753.376104:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760753.376127:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25686016, count: 4096 00000080:00200000:2.0:1789760753.376134:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25686016, bytes: 4096 00000080:00200000:2.0:1789760753.376144:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760753.376152:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760753.376154:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25686016, 25690112) 0 0 00000080:00200000:2.0:1789760753.376157:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760753.376171:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6271, 6271] 00020000:00200000:2.0:1789760753.376178:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25686016 chunk: [25686016, 25690112) 25690112, 4096 00020000:00200000:2.0:1789760753.376181:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760753.376196:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6271, 6271] 00000008:00010000:2.0:1789760753.376209:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760753.376213:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760753.376221:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25686016, 25690112) append 0 append lockpos 0 00000080:00200000:2.0:1789760753.376231:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25686016=0x187f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760753.376279:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760753.376282:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25690112 00000020:00010000:2.0:1789760753.376288:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760753.376291:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.376294:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.376297:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760753.376300:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760753.376307:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760753.376312:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760753.376331:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760753.376437:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 25686016<>25690112 00000100:00100000:3.0:1789760753.376511:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512664064:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760753.416323:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973590 gen 3 00000008:00010000:3.0:1789760753.416368:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760753.416374:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760753.416382:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760753.416455:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760753.416482:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512664064:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760753.416625:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6271, 6271] 00000080:00200000:2.0:1789760753.416641:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760753.416647:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760753.416658:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25690112 i_blocks 51207 00000020:00200000:2.0:1789760753.416661:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760753.416670:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760753.416694:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25690112, count: 4096 00000080:00200000:2.0:1789760753.416700:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25690112, bytes: 4096 00000080:00200000:2.0:1789760753.416708:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760753.416716:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760753.416717:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25690112, 25694208) 0 0 00000080:00200000:2.0:1789760753.416721:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760753.416734:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6272, 6272] 00020000:00200000:2.0:1789760753.416741:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25690112 chunk: [25690112, 25694208) 25694208, 4096 00020000:00200000:2.0:1789760753.416744:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760753.416758:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6272, 6272] 00000008:00010000:2.0:1789760753.416773:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760753.416778:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760753.416787:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25690112, 25694208) append 0 append lockpos 0 00000080:00200000:2.0:1789760753.416798:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25690112=0x1880000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760753.416840:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760753.416843:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25694208 00000020:00010000:2.0:1789760753.416849:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760753.416853:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.416855:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.416858:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760753.416861:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760753.416868:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760753.416873:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760753.416886:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760753.416987:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 25690112<>25694208 00000100:00100000:3.0:1789760753.417142:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760753.417155:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512664192:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760753.437177:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760753.458960:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973591 gen 3 00000008:00010000:3.0:1789760753.458993:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760753.459000:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760753.459017:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760753.459033:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760753.459058:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512664192:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760753.459155:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6272, 6272] 00000080:00200000:2.0:1789760753.459167:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760753.459171:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760753.459183:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25694208 i_blocks 51207 00000020:00200000:2.0:1789760753.459186:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760753.459194:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760753.459220:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25694208, count: 4096 00000080:00200000:2.0:1789760753.459226:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25694208, bytes: 4096 00000080:00200000:2.0:1789760753.459234:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760753.459241:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760753.459243:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25694208, 25698304) 0 0 00000080:00200000:2.0:1789760753.459247:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760753.459259:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6273, 6273] 00020000:00200000:2.0:1789760753.459265:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25694208 chunk: [25694208, 25698304) 25698304, 4096 00020000:00200000:2.0:1789760753.459269:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760753.459282:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6273, 6273] 00000008:00010000:2.0:1789760753.459294:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760753.459299:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760753.459307:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25694208, 25698304) append 0 append lockpos 0 00000080:00200000:2.0:1789760753.459317:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25694208=0x1881000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760753.459364:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760753.459367:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25698304 00000020:00010000:2.0:1789760753.459373:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760753.459377:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.459380:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.459383:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760753.459385:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760753.459391:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760753.459396:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760753.459408:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760753.459488:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 25694208<>25698304 00000100:00100000:3.0:1789760753.459539:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512664320:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760753.504392:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973592 gen 3 00000008:00010000:3.0:1789760753.504429:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760753.504435:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760753.504443:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760753.504465:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760753.504487:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512664320:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760753.504835:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6273, 6273] 00000080:00200000:2.0:1789760753.504846:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760753.504851:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760753.504862:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25698304 i_blocks 51207 00000020:00200000:2.0:1789760753.504865:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760753.504872:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760753.504895:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25698304, count: 4096 00000080:00200000:2.0:1789760753.504917:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25698304, bytes: 4096 00000080:00200000:2.0:1789760753.504925:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760753.504932:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760753.504934:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25698304, 25702400) 0 0 00000080:00200000:2.0:1789760753.504938:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760753.504949:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6274, 6274] 00020000:00200000:2.0:1789760753.504955:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25698304 chunk: [25698304, 25702400) 25702400, 4096 00020000:00200000:2.0:1789760753.504959:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760753.504971:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6274, 6274] 00000008:00010000:2.0:1789760753.504984:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760753.504989:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760753.504997:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25698304, 25702400) append 0 append lockpos 0 00000080:00200000:2.0:1789760753.505008:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25698304=0x1882000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760753.505045:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760753.505047:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25702400 00000020:00010000:2.0:1789760753.506228:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760753.506235:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.506239:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.506242:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760753.506245:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760753.506254:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760753.506262:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760753.506284:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760753.506376:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 25698304<>25702400 00000100:00100000:3.0:1789760753.506550:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512664448:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760753.556591:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760753.556602:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760753.556609:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760753.556622:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000080:00200000:2.0:1789760753.556647:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6274, 6274] 00000080:00200000:2.0:1789760753.556661:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760753.556666:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760753.556677:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25702400 i_blocks 51207 00000020:00200000:2.0:1789760753.556680:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760753.556687:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760753.556714:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25702400, count: 4096 00000100:00100000:3.0:1789760753.556716:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512664448:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760753.556720:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25702400, bytes: 4096 00000080:00200000:2.0:1789760753.556727:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760753.556733:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760753.556735:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25702400, 25706496) 0 0 00000080:00200000:2.0:1789760753.556738:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760753.556752:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6275, 6275] 00020000:00200000:2.0:1789760753.556759:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25702400 chunk: [25702400, 25706496) 25706496, 4096 00020000:00200000:2.0:1789760753.556763:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760753.556777:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6275, 6275] 00000008:00010000:2.0:1789760753.556791:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760753.556796:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760753.556805:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25702400, 25706496) append 0 append lockpos 0 00000080:00200000:2.0:1789760753.556816:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25702400=0x1883000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760753.556858:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760753.556861:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25706496 00000020:00010000:2.0:1789760753.556867:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760753.556870:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.556873:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.556875:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760753.556877:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760753.556883:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760753.556888:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760753.556998:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760753.557087:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 25702400<>25706496 00000100:00100000:2.0:1789760753.557116:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760753.557128:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512664576:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760753.607097:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973594 gen 3 00000008:00010000:3.0:1789760753.607148:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760753.607155:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760753.607162:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760753.607181:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760753.607204:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512664576:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760753.607278:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6275, 6275] 00000080:00200000:2.0:1789760753.607295:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760753.607300:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760753.607314:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25706496 i_blocks 51207 00000020:00200000:2.0:1789760753.607317:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760753.607325:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760753.607399:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25706496, count: 4096 00000080:00200000:2.0:1789760753.607407:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25706496, bytes: 4096 00000080:00200000:2.0:1789760753.607415:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760753.607421:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760753.607423:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25706496, 25710592) 0 0 00000080:00200000:2.0:1789760753.607427:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760753.607439:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6276, 6276] 00020000:00200000:2.0:1789760753.607446:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25706496 chunk: [25706496, 25710592) 25710592, 4096 00020000:00200000:2.0:1789760753.607449:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760753.607464:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6276, 6276] 00000008:00010000:2.0:1789760753.607478:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760753.607483:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760753.607492:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25706496, 25710592) append 0 append lockpos 0 00000080:00200000:2.0:1789760753.607502:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25706496=0x1884000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760753.607543:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760753.607545:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25710592 00000020:00010000:2.0:1789760753.607552:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760753.607556:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.607559:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.607562:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760753.607565:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760753.607572:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760753.607577:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760753.607590:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760753.607673:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 25706496<>25710592 00000100:00100000:3.0:1789760753.607821:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760753.607835:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512664704:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760753.645734:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760753.645746:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760753.645754:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760753.645808:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760753.645833:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512664704:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760753.648803:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6276, 6276] 00000080:00200000:2.0:1789760753.648889:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760753.648895:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760753.648923:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25710592 i_blocks 51207 00000020:00200000:2.0:1789760753.648928:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760753.648941:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760753.648967:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25710592, count: 4096 00000080:00200000:2.0:1789760753.648974:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25710592, bytes: 4096 00000080:00200000:2.0:1789760753.648986:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760753.648993:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760753.648996:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25710592, 25714688) 0 0 00000080:00200000:2.0:1789760753.648999:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760753.649012:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6277, 6277] 00020000:00200000:2.0:1789760753.649019:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25710592 chunk: [25710592, 25714688) 25714688, 4096 00020000:00200000:2.0:1789760753.649023:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760753.649039:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6277, 6277] 00000008:00010000:2.0:1789760753.649055:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760753.649060:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760753.649069:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25710592, 25714688) append 0 append lockpos 0 00000080:00200000:2.0:1789760753.649082:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25710592=0x1885000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760753.649138:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760753.649142:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25714688 00000020:00010000:2.0:1789760753.649148:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760753.649151:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.649154:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.649157:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760753.649161:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760753.649168:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760753.649174:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760753.649200:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760753.649432:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 25710592<>25714688 00000100:00100000:3.0:1789760753.649517:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512664832:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760753.693094:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973596 gen 3 00000008:00010000:3.0:1789760753.693144:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760753.693151:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760753.693159:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760753.693175:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760753.693197:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512664832:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760753.693245:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6277, 6277] 00000080:00200000:2.0:1789760753.693257:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760753.693261:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760753.693272:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25714688 i_blocks 51207 00000020:00200000:2.0:1789760753.693279:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760753.693287:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760753.693313:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25714688, count: 4096 00000080:00200000:2.0:1789760753.693318:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25714688, bytes: 4096 00000080:00200000:2.0:1789760753.693345:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760753.693353:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760753.693355:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25714688, 25718784) 0 0 00000080:00200000:2.0:1789760753.693359:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760753.693380:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6278, 6278] 00020000:00200000:2.0:1789760753.693387:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25714688 chunk: [25714688, 25718784) 25718784, 4096 00020000:00200000:2.0:1789760753.693392:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760753.693407:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6278, 6278] 00000008:00010000:2.0:1789760753.693421:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760753.693425:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760753.693433:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25714688, 25718784) append 0 append lockpos 0 00000080:00200000:2.0:1789760753.693443:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25714688=0x1886000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760753.693493:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760753.693495:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25718784 00000020:00010000:2.0:1789760753.693649:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760753.693654:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.693657:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.693660:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760753.693662:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760753.693670:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760753.693676:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760753.693692:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760753.693758:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 25714688<>25718784 00000100:00100000:3.0:1789760753.693800:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760753.693823:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512664960:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760753.739705:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973597 gen 3 00000008:00010000:3.0:1789760753.739999:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760753.740419:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760753.740428:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760753.740449:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760753.740531:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512664960:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760753.740533:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6278, 6278] 00000080:00200000:2.0:1789760753.740550:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760753.740555:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760753.740568:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25718784 i_blocks 51207 00000020:00200000:2.0:1789760753.740572:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760753.740582:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760753.740609:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25718784, count: 4096 00000080:00200000:2.0:1789760753.740616:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25718784, bytes: 4096 00000080:00200000:2.0:1789760753.740625:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760753.740633:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760753.740635:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25718784, 25722880) 0 0 00000080:00200000:2.0:1789760753.740639:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760753.740653:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6279, 6279] 00020000:00200000:2.0:1789760753.740659:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25718784 chunk: [25718784, 25722880) 25722880, 4096 00020000:00200000:2.0:1789760753.740663:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760753.740679:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6279, 6279] 00000008:00010000:2.0:1789760753.740692:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760753.740698:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760753.740707:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25718784, 25722880) append 0 append lockpos 0 00000080:00200000:2.0:1789760753.740717:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25718784=0x1887000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760753.740759:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760753.740762:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25722880 00000020:00010000:2.0:1789760753.740769:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760753.740773:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.740776:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.740779:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760753.740782:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760753.740789:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760753.740795:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760753.740809:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760753.742382:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 25718784<>25722880 00000100:00100000:2.0:1789760753.742524:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760753.742543:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512665088:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760753.789273:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973598 gen 3 00000008:00010000:3.0:1789760753.789311:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760753.789316:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760753.789324:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760753.789348:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760753.789372:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512665088:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760753.789735:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6279, 6279] 00000080:00200000:2.0:1789760753.789748:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760753.789753:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760753.789764:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25722880 i_blocks 51207 00000020:00200000:2.0:1789760753.789768:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760753.789775:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760753.789800:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25722880, count: 4096 00000080:00200000:2.0:1789760753.789808:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25722880, bytes: 4096 00000080:00200000:2.0:1789760753.789816:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760753.789823:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760753.789825:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25722880, 25726976) 0 0 00000080:00200000:2.0:1789760753.789829:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760753.789842:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6280, 6280] 00020000:00200000:2.0:1789760753.789848:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25722880 chunk: [25722880, 25726976) 25726976, 4096 00020000:00200000:2.0:1789760753.789851:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760753.790599:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6280, 6280] 00000008:00010000:2.0:1789760753.790617:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760753.790623:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760753.790633:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25722880, 25726976) append 0 append lockpos 0 00000080:00200000:2.0:1789760753.790643:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25722880=0x1888000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760753.790688:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760753.790691:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25726976 00000020:00010000:2.0:1789760753.790697:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760753.790701:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.790704:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.790707:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760753.790710:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760753.790716:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760753.790722:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760753.790736:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760753.790811:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 25722880<>25726976 00000100:00100000:3.0:1789760753.791016:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512665216:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760753.835576:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973599 gen 3 00000008:00010000:3.0:1789760753.835616:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760753.835623:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760753.835632:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760753.835650:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760753.835674:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512665216:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760753.835999:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6280, 6280] 00000080:00200000:2.0:1789760753.836019:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760753.836025:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760753.836036:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25726976 i_blocks 51207 00000020:00200000:2.0:1789760753.836040:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760753.836047:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760753.836069:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25726976, count: 4096 00000080:00200000:2.0:1789760753.836075:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25726976, bytes: 4096 00000080:00200000:2.0:1789760753.836082:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760753.836088:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760753.836090:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25726976, 25731072) 0 0 00000080:00200000:2.0:1789760753.836094:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760753.836106:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6281, 6281] 00020000:00200000:2.0:1789760753.836113:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25726976 chunk: [25726976, 25731072) 25731072, 4096 00020000:00200000:2.0:1789760753.836117:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760753.836131:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6281, 6281] 00000008:00010000:2.0:1789760753.836143:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760753.836148:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760753.836156:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25726976, 25731072) append 0 append lockpos 0 00000080:00200000:2.0:1789760753.836165:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25726976=0x1889000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760753.836204:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760753.836206:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25731072 00000020:00010000:2.0:1789760753.836212:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760753.836215:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.836218:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.836221:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760753.836224:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760753.836230:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760753.836234:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760753.836246:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000008:00100000:2.0:1789760753.836311:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 25726976<>25731072 00000100:00100000:3.0:1789760753.836497:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512665344:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760753.881874:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760753.881884:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760753.881893:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760753.882086:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760753.882112:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512665344:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760753.882236:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6281, 6281] 00000080:00200000:2.0:1789760753.882249:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760753.882254:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760753.882265:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25731072 i_blocks 51207 00000020:00200000:2.0:1789760753.882269:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760753.882276:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760753.882304:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25731072, count: 4096 00000080:00200000:2.0:1789760753.882311:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25731072, bytes: 4096 00000080:00200000:2.0:1789760753.882318:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760753.882326:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760753.882328:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25731072, 25735168) 0 0 00000080:00200000:2.0:1789760753.882341:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760753.882356:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6282, 6282] 00020000:00200000:2.0:1789760753.882362:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25731072 chunk: [25731072, 25735168) 25735168, 4096 00020000:00200000:2.0:1789760753.882366:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760753.882380:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6282, 6282] 00000008:00010000:2.0:1789760753.882393:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760753.882399:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760753.882407:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25731072, 25735168) append 0 append lockpos 0 00000080:00200000:2.0:1789760753.882490:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25731072=0x188a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760753.882536:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760753.882538:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25735168 00000020:00010000:2.0:1789760753.882545:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760753.882549:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.882552:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.882555:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760753.882558:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760753.882563:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760753.882570:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760753.882583:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760753.882942:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 25731072<>25735168 00000100:00100000:3.0:1789760753.883015:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512665472:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760753.930600:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973601 gen 3 00000008:00010000:3.0:1789760753.930650:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760753.930657:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760753.930665:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760753.930683:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760753.930708:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512665472:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760753.930927:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6282, 6282] 00000080:00200000:2.0:1789760753.930940:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760753.930946:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760753.930959:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25735168 i_blocks 51207 00000020:00200000:2.0:1789760753.930962:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760753.930970:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760753.930994:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25735168, count: 4096 00000080:00200000:2.0:1789760753.931000:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25735168, bytes: 4096 00000080:00200000:2.0:1789760753.931017:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760753.931024:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760753.931026:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25735168, 25739264) 0 0 00000080:00200000:2.0:1789760753.931030:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760753.931044:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6283, 6283] 00020000:00200000:2.0:1789760753.931051:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25735168 chunk: [25735168, 25739264) 25739264, 4096 00020000:00200000:2.0:1789760753.931055:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760753.931072:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6283, 6283] 00000008:00010000:2.0:1789760753.931086:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760753.931091:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760753.931099:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25735168, 25739264) append 0 append lockpos 0 00000080:00200000:2.0:1789760753.931111:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25735168=0x188b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760753.931155:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760753.931158:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25739264 00000020:00010000:2.0:1789760753.931165:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760753.931169:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.931172:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760753.931175:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760753.931178:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760753.931184:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760753.931189:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760753.931203:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760753.931281:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 25735168<>25739264 00000100:00100000:3.0:1789760753.931456:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512665600:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760754.000923:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760754.000935:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760754.000945:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760754.000962:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760754.000984:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512665600:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760754.001116:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6283, 6283] 00000080:00200000:2.0:1789760754.001130:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760754.001135:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760754.001146:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25739264 i_blocks 51207 00000020:00200000:2.0:1789760754.001150:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760754.001157:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760754.001182:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25739264, count: 4096 00000080:00200000:2.0:1789760754.001189:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25739264, bytes: 4096 00000080:00200000:2.0:1789760754.001197:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760754.001204:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760754.001206:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25739264, 25743360) 0 0 00000080:00200000:2.0:1789760754.001210:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760754.001223:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6284, 6284] 00020000:00200000:2.0:1789760754.001229:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25739264 chunk: [25739264, 25743360) 25743360, 4096 00020000:00200000:2.0:1789760754.001233:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760754.001246:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6284, 6284] 00000008:00010000:2.0:1789760754.001258:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760754.001489:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760754.001500:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25739264, 25743360) append 0 append lockpos 0 00000080:00200000:2.0:1789760754.001512:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25739264=0x188c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760754.001556:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760754.001559:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25743360 00000020:00010000:2.0:1789760754.001565:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760754.001569:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.001571:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.001574:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760754.001577:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760754.001583:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760754.001589:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760754.001602:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000008:00100000:2.0:1789760754.001679:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 25739264<>25743360 00000100:00100000:3.0:1789760754.001737:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512665728:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760754.054705:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973602 gen 3 00000100:00100000:3.0:1789760754.054805:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368c000 x1876699512665728/t4294973603(4294973603) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760770 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760754.054854:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760754.054862:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760754.054872:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760754.054898:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760754.054955:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512665728:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760754.055973:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6284, 6284] 00000080:00200000:2.0:1789760754.055993:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760754.056000:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760754.056014:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25743360 i_blocks 51207 00000020:00200000:2.0:1789760754.056019:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760754.056029:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760754.056065:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25743360, count: 4096 00000080:00200000:2.0:1789760754.056073:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25743360, bytes: 4096 00000080:00200000:2.0:1789760754.056086:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760754.056095:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760754.056097:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25743360, 25747456) 0 0 00000080:00200000:2.0:1789760754.056102:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760754.056120:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6285, 6285] 00020000:00200000:2.0:1789760754.056128:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25743360 chunk: [25743360, 25747456) 25747456, 4096 00020000:00200000:2.0:1789760754.056133:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760754.056153:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6285, 6285] 00000008:00010000:2.0:1789760754.056171:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760754.056178:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760754.056189:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25743360, 25747456) append 0 append lockpos 0 00000080:00200000:2.0:1789760754.057542:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25743360=0x188d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760754.057647:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760754.057650:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25747456 00000020:00010000:2.0:1789760754.057662:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760754.057666:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.057669:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc018e42d88/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.057673:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760754.057676:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760754.057686:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760754.057694:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760754.058963:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000008:00100000:2.0:1789760754.059097:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 25743360<>25747456 00000100:00100000:2.0:1789760754.059169:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512665856:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760754.139147:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973604 gen 3 00000008:00010000:3.0:1789760754.139208:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760754.139217:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760754.139227:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760754.139448:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760754.139479:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512665856:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760754.145647:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512665984:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760754.187504:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973605 gen 3 00000008:00010000:1.0:1789760754.187559:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760754.187567:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760754.187579:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760754.187606:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760754.187639:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512665984:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760754.188176:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6287, 6287] 00020000:00200000:1.0:1789760754.188186:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25751552 chunk: [25751552, 25755648) 25755648, 4096 00020000:00200000:1.0:1789760754.188190:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760754.188211:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6287, 6287] 00000008:00010000:1.0:1789760754.188229:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760754.188236:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760754.188248:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25751552, 25755648) append 0 append lockpos 0 00000080:00200000:1.0:1789760754.188260:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25751552=0x188f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760754.188313:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760754.188315:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25755648 00000020:00010000:1.0:1789760754.188323:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760754.188326:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.188340:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.188344:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760754.188347:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760754.188354:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760754.188360:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760754.188372:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760754.188465:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94700 - object 0x240000400:5 offset 25751552<>25755648 00000080:00200000:1.0:1789760754.233699:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6287, 6287] 00000080:00200000:1.0:1789760754.233715:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760754.233783:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760754.233872:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25755648 i_blocks 51207 00000020:00200000:1.0:1789760754.233877:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760754.233888:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760754.233961:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25755648, count: 4096 00000080:00200000:1.0:1789760754.233969:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25755648, bytes: 4096 00000080:00200000:1.0:1789760754.233980:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760754.233989:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760754.233991:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25755648, 25759744) 0 0 00000080:00200000:1.0:1789760754.233997:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760754.234086:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6288, 6288] 00020000:00200000:1.0:1789760754.234095:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25755648 chunk: [25755648, 25759744) 25759744, 4096 00020000:00200000:1.0:1789760754.234100:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760754.234120:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6288, 6288] 00000008:00010000:1.0:1789760754.234138:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760754.234143:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760754.234154:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25755648, 25759744) append 0 append lockpos 0 00000080:00200000:1.0:1789760754.234167:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25755648=0x1890000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760754.234224:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760754.234226:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25759744 00000020:00010000:1.0:1789760754.234235:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760754.234238:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.234242:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.234245:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760754.234248:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760754.234255:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760754.234262:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760754.234280:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760754.234398:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 25755648<>25759744 00000080:00200000:2.0:1789760754.275575:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6288, 6288] 00000080:00200000:2.0:1789760754.275593:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760754.275703:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760754.275776:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25759744 i_blocks 51207 00000020:00200000:2.0:1789760754.275782:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760754.276404:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760754.276441:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25759744, count: 4096 00000080:00200000:2.0:1789760754.276450:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25759744, bytes: 4096 00000080:00200000:2.0:1789760754.276461:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760754.276470:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760754.276472:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25759744, 25763840) 0 0 00000080:00200000:2.0:1789760754.276476:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760754.276493:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6289, 6289] 00020000:00200000:2.0:1789760754.276503:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25759744 chunk: [25759744, 25763840) 25763840, 4096 00020000:00200000:2.0:1789760754.276507:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760754.276529:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6289, 6289] 00000008:00010000:2.0:1789760754.276548:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760754.276554:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760754.276565:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25759744, 25763840) append 0 append lockpos 0 00000080:00200000:2.0:1789760754.276579:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25759744=0x1891000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760754.276965:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760754.276969:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25763840 00000020:00010000:2.0:1789760754.276978:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760754.276982:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.276985:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.276988:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760754.276991:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760754.277063:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760754.277071:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760754.277096:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760754.277198:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 25759744<>25763840 00000100:00100000:3.0:1789760754.277363:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760754.277463:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512666368:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760754.320996:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973608 gen 3 00000008:00010000:3.0:1789760754.321138:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760754.321146:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760754.321157:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760754.321175:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760754.321205:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512666368:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760754.321385:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6289, 6289] 00000080:00200000:2.0:1789760754.321404:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760754.321410:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760754.321426:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25763840 i_blocks 51207 00000020:00200000:2.0:1789760754.321430:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760754.321439:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760754.321474:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25763840, count: 4096 00000080:00200000:2.0:1789760754.321482:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25763840, bytes: 4096 00000080:00200000:2.0:1789760754.321493:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760754.321503:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760754.321505:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25763840, 25767936) 0 0 00000080:00200000:2.0:1789760754.321509:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760754.321527:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6290, 6290] 00020000:00200000:2.0:1789760754.321536:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25763840 chunk: [25763840, 25767936) 25767936, 4096 00020000:00200000:2.0:1789760754.321541:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760754.321563:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6290, 6290] 00000008:00010000:2.0:1789760754.321582:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760754.321589:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760754.321614:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25763840, 25767936) append 0 append lockpos 0 00000080:00200000:2.0:1789760754.321773:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25763840=0x1892000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760754.321846:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760754.321849:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25767936 00000020:00010000:2.0:1789760754.321859:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760754.321863:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.321866:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760754.321870:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760754.321873:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760754.321881:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760754.321888:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760754.321945:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760754.322134:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 25763840<>25767936 00000100:00100000:2.0:1789760754.322181:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760754.322199:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512666496:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760754.380386:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973609 gen 3 00000008:00010000:3.0:1789760754.380438:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760754.380446:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760754.380457:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760754.380487:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760754.380523:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512666496:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760754.437030:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6291, 6291] 00000080:00200000:3.0:1789760754.437049:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.437125:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.437140:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25772032 i_blocks 51207 00000020:00200000:3.0:1789760754.437144:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.437154:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.437189:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25772032, count: 4096 00000080:00200000:3.0:1789760754.437197:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25772032, bytes: 4096 00000080:00200000:3.0:1789760754.437207:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.437216:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.437218:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25772032, 25776128) 0 0 00000080:00200000:3.0:1789760754.437222:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.437239:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6292, 6292] 00020000:00200000:3.0:1789760754.437248:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25772032 chunk: [25772032, 25776128) 25776128, 4096 00020000:00200000:3.0:1789760754.437252:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.437272:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6292, 6292] 00000008:00010000:3.0:1789760754.437290:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.437295:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760754.437306:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25772032, 25776128) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.437320:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25772032=0x1894000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.437382:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.437385:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25776128 00000020:00010000:3.0:1789760754.437392:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.437396:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.437398:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.437402:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.437406:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.437413:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.437419:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1789760754.461560:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512667008:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760754.461834:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1789760754.461842:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e6cf7267-9b19-462a-a263-14115df15dab:2361:1876699512666880:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760754.461882:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2361:1876699512667136:192.168.203.156@tcp:400:kworker.0 00010000:00010000:3.0:1789760754.462567:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760754.467130:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e6cf7267-9b19-462a-a263-14115df15dab:2361:1876699512666880:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760754.470558:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2361:1876699512667136:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760754.470576:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512667008:192.168.203.156@tcp:400:kworker.0 00000080:00200000:3.0:1789760754.479436:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6292, 6292] 00000080:00200000:3.0:1789760754.479453:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.479459:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.479475:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25776128 i_blocks 51207 00000020:00200000:3.0:1789760754.479478:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.479488:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.479521:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25776128, count: 4096 00000080:00200000:3.0:1789760754.479528:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25776128, bytes: 4096 00000080:00200000:3.0:1789760754.479540:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.479549:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.479551:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25776128, 25780224) 0 0 00000080:00200000:3.0:1789760754.479556:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.479572:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6293, 6293] 00020000:00200000:3.0:1789760754.479580:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25776128 chunk: [25776128, 25780224) 25780224, 4096 00020000:00200000:3.0:1789760754.479586:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.479605:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6293, 6293] 00000008:00010000:3.0:1789760754.479622:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.479628:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760754.479639:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25776128, 25780224) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.479653:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25776128=0x1895000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.479710:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.479713:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25780224 00000020:00010000:3.0:1789760754.479721:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.479725:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.479729:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.479733:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.479736:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.479744:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.479751:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760754.479766:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760754.479869:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae300 - object 0x240000400:5 offset 25776128<>25780224 00000100:00100000:2.0:1789760754.480038:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512667264:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760754.528999:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973611 gen 3 00000100:00100000:2.0:1789760754.529048:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc0107ae300 x1876699512667264/t4294973612(4294973612) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760770 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760754.529101:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760754.529109:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.529120:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.529141:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760754.529175:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512667264:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760754.530799:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6293, 6293] 00000080:00200000:3.0:1789760754.530823:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.530831:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.530848:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25780224 i_blocks 51207 00000020:00200000:3.0:1789760754.530854:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.530869:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.530954:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25780224, count: 4096 00000080:00200000:3.0:1789760754.530963:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25780224, bytes: 4096 00000080:00200000:3.0:1789760754.530978:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.530987:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.530990:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25780224, 25784320) 0 0 00000080:00200000:3.0:1789760754.530995:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.531012:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6294, 6294] 00020000:00200000:3.0:1789760754.531021:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25780224 chunk: [25780224, 25784320) 25784320, 4096 00020000:00200000:3.0:1789760754.531027:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.531048:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6294, 6294] 00000008:00010000:3.0:1789760754.531067:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.531073:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760754.531086:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25780224, 25784320) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.531102:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25780224=0x1896000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.531172:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.531176:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25784320 00000020:00010000:3.0:1789760754.531185:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.531188:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.531191:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.531195:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.531198:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.531206:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.531214:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760754.531237:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760754.531365:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af480 - object 0x240000400:5 offset 25780224<>25784320 00000100:00100000:2.0:1789760754.531573:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512667392:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760754.586286:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973612 gen 3 00000100:00100000:2.0:1789760754.586337:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc0107af480 x1876699512667392/t4294973613(4294973613) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760770 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760754.586387:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760754.586394:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.586405:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.586425:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760754.586449:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512667392:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760754.588242:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6294, 6294] 00000080:00200000:3.0:1789760754.588267:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.588274:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.588293:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25784320 i_blocks 51207 00000020:00200000:3.0:1789760754.588299:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.588313:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.588358:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25784320, count: 4096 00000080:00200000:3.0:1789760754.588366:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25784320, bytes: 4096 00000080:00200000:3.0:1789760754.588381:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.588391:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.588393:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25784320, 25788416) 0 0 00000080:00200000:3.0:1789760754.588398:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.588415:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6295, 6295] 00020000:00200000:3.0:1789760754.588425:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25784320 chunk: [25784320, 25788416) 25788416, 4096 00020000:00200000:3.0:1789760754.588429:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.588451:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6295, 6295] 00000008:00010000:3.0:1789760754.588469:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.588475:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760754.588488:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25784320, 25788416) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.588503:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25784320=0x1897000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.588574:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.588576:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25788416 00000020:00010000:3.0:1789760754.588585:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.588589:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.588593:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.588597:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.588600:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.588607:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.588615:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760754.588635:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760754.588822:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac380 - object 0x240000400:5 offset 25784320<>25788416 00000100:00100000:2.0:1789760754.588943:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512667520:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760754.641518:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973614 gen 3 00000008:00010000:2.0:1789760754.641588:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760754.641596:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.641607:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.641627:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760754.641658:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512667520:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760754.641842:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6295, 6295] 00000080:00200000:3.0:1789760754.641859:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.641865:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.641880:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25788416 i_blocks 51207 00000020:00200000:3.0:1789760754.641884:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.641893:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.641965:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25788416, count: 4096 00000080:00200000:3.0:1789760754.641973:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25788416, bytes: 4096 00000080:00200000:3.0:1789760754.641984:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.642003:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.642006:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25788416, 25792512) 0 0 00000080:00200000:3.0:1789760754.642011:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.642029:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6296, 6296] 00020000:00200000:3.0:1789760754.642038:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25788416 chunk: [25788416, 25792512) 25792512, 4096 00020000:00200000:3.0:1789760754.642043:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.642064:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6296, 6296] 00000008:00010000:3.0:1789760754.642080:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.642086:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760754.642099:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25788416, 25792512) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.642111:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25788416=0x1898000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.642162:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.642165:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25792512 00000020:00010000:3.0:1789760754.642174:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.642177:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.642180:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.642184:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.642186:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.642194:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.642201:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760754.642218:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760754.642320:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad180 - object 0x240000400:5 offset 25788416<>25792512 00000100:00100000:2.0:1789760754.642528:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512667648:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760754.695618:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760754.695635:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.695658:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.695680:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000080:00200000:3.0:1789760754.695707:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6296, 6296] 00000100:00100000:2.0:1789760754.695716:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512667648:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760754.695726:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.695732:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.695746:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25792512 i_blocks 51207 00000020:00200000:3.0:1789760754.695750:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.695760:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.695809:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25792512, count: 4096 00000080:00200000:3.0:1789760754.695818:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25792512, bytes: 4096 00000080:00200000:3.0:1789760754.695827:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.695836:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.695837:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25792512, 25796608) 0 0 00000080:00200000:3.0:1789760754.695841:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.695858:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6297, 6297] 00020000:00200000:3.0:1789760754.695866:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25792512 chunk: [25792512, 25796608) 25796608, 4096 00020000:00200000:3.0:1789760754.695870:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.695888:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6297, 6297] 00000008:00010000:3.0:1789760754.695940:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.695946:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760754.695959:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25792512, 25796608) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.695974:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25792512=0x1899000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.696224:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.696228:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25796608 00000020:00010000:3.0:1789760754.696237:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.696241:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.696244:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.696248:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.696251:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.696259:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.696266:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760754.696281:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760754.696402:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adf80 - object 0x240000400:5 offset 25792512<>25796608 00000100:00100000:2.0:1789760754.696595:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512667776:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760754.743470:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973616 gen 3 00000008:00010000:2.0:1789760754.743535:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760754.743544:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.743555:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.743723:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760754.743755:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512667776:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760754.744243:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6297, 6297] 00000080:00200000:3.0:1789760754.744261:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.744267:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.744282:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25796608 i_blocks 51207 00000020:00200000:3.0:1789760754.744287:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.744298:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.744345:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25796608, count: 4096 00000080:00200000:3.0:1789760754.744354:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25796608, bytes: 4096 00000080:00200000:3.0:1789760754.744366:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.744376:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.744378:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25796608, 25800704) 0 0 00000080:00200000:3.0:1789760754.744382:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.744399:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6298, 6298] 00020000:00200000:3.0:1789760754.744409:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25796608 chunk: [25796608, 25800704) 25800704, 4096 00020000:00200000:3.0:1789760754.744414:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.744433:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6298, 6298] 00000008:00010000:3.0:1789760754.744450:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.744457:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760754.744469:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25796608, 25800704) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.744484:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25796608=0x189a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.744544:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.744548:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25800704 00000020:00010000:3.0:1789760754.744556:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.744560:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.744563:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.744568:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.744570:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.744579:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.744586:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760754.744606:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760754.744777:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aea00 - object 0x240000400:5 offset 25796608<>25800704 00000100:00100000:2.0:1789760754.744855:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512667904:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760754.788723:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760754.788738:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.788751:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.788844:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000080:00200000:1.0:1789760754.788851:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6298, 6298] 00000080:00200000:1.0:1789760754.788876:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:2.0:1789760754.788881:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512667904:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760754.788891:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760754.788940:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25800704 i_blocks 51207 00000020:00200000:1.0:1789760754.788944:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760754.788961:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760754.788999:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25800704, count: 4096 00000080:00200000:1.0:1789760754.789012:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25800704, bytes: 4096 00000080:00200000:1.0:1789760754.789030:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760754.789038:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760754.789041:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25800704, 25804800) 0 0 00000080:00200000:1.0:1789760754.789048:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760754.789065:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6299, 6299] 00020000:00200000:1.0:1789760754.789073:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25800704 chunk: [25800704, 25804800) 25804800, 4096 00020000:00200000:1.0:1789760754.789078:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760754.789108:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6299, 6299] 00000008:00010000:1.0:1789760754.789126:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760754.789138:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760754.789150:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25800704, 25804800) append 0 append lockpos 0 00000080:00200000:1.0:1789760754.789168:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25800704=0x189b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760754.789234:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760754.789236:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25804800 00000020:00010000:1.0:1789760754.789245:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760754.789248:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.789251:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.789259:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760754.789262:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760754.789270:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760754.789276:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760754.789297:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760754.789413:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 25800704<>25804800 00000080:00200000:1.0:1789760754.824766:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6299, 6299] 00000080:00200000:1.0:1789760754.824783:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760754.824800:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760754.824815:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25804800 i_blocks 51207 00000020:00200000:1.0:1789760754.824819:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760754.824830:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760754.824875:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25804800, count: 4096 00000080:00200000:1.0:1789760754.824883:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25804800, bytes: 4096 00000080:00200000:1.0:1789760754.824893:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760754.824938:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760754.824941:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25804800, 25808896) 0 0 00000080:00200000:1.0:1789760754.824946:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760754.824963:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6300, 6300] 00020000:00200000:1.0:1789760754.824972:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25804800 chunk: [25804800, 25808896) 25808896, 4096 00020000:00200000:1.0:1789760754.824977:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760754.825003:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6300, 6300] 00000008:00010000:1.0:1789760754.825020:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760754.825025:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760754.825036:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25804800, 25808896) append 0 append lockpos 0 00000080:00200000:1.0:1789760754.825049:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25804800=0x189c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760754.825109:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760754.825113:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25808896 00000020:00010000:1.0:1789760754.825121:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760754.825124:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.825126:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760754.825130:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760754.825133:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760754.825140:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760754.825147:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760754.825165:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760754.825282:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95c00 - object 0x240000400:5 offset 25804800<>25808896 00000080:00200000:3.0:1789760754.890202:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6300, 6300] 00000080:00200000:3.0:1789760754.890219:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.890225:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.890241:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25808896 i_blocks 51207 00000020:00200000:3.0:1789760754.890245:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.890256:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.890294:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25808896, count: 4096 00000080:00200000:3.0:1789760754.890301:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25808896, bytes: 4096 00000080:00200000:3.0:1789760754.890317:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.890335:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.890337:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25808896, 25812992) 0 0 00000080:00200000:3.0:1789760754.890342:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.890358:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6301, 6301] 00020000:00200000:3.0:1789760754.890367:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25808896 chunk: [25808896, 25812992) 25812992, 4096 00020000:00200000:3.0:1789760754.890371:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.890392:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6301, 6301] 00000008:00010000:3.0:1789760754.890411:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.890417:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760754.890429:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25808896, 25812992) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.890444:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25808896=0x189d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.890519:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.890522:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25812992 00000020:00010000:3.0:1789760754.890531:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.890534:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.890538:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.890543:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.890546:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.890554:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.890562:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760754.890579:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760754.890687:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af100 - object 0x240000400:5 offset 25808896<>25812992 00000100:00100000:2.0:1789760754.891132:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512668288:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760754.984983:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973620 gen 3 00000008:00010000:2.0:1789760754.985039:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760754.985049:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760754.985062:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760754.985094:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760754.985124:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512668288:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760754.985425:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6301, 6301] 00000080:00200000:3.0:1789760754.985442:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760754.985448:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760754.985464:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25812992 i_blocks 51207 00000020:00200000:3.0:1789760754.985469:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760754.985478:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760754.985515:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25812992, count: 4096 00000080:00200000:3.0:1789760754.985523:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25812992, bytes: 4096 00000080:00200000:3.0:1789760754.985536:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760754.985546:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760754.985549:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25812992, 25817088) 0 0 00000080:00200000:3.0:1789760754.985553:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760754.985572:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6302, 6302] 00020000:00200000:3.0:1789760754.985582:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25812992 chunk: [25812992, 25817088) 25817088, 4096 00020000:00200000:3.0:1789760754.985587:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760754.985608:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6302, 6302] 00000008:00010000:3.0:1789760754.985627:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760754.985633:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760754.985644:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25812992, 25817088) append 0 append lockpos 0 00000080:00200000:3.0:1789760754.985658:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25812992=0x189e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760754.985861:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760754.985865:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25817088 00000020:00010000:3.0:1789760754.985874:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760754.985877:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.985880:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760754.985884:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760754.985887:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760754.985895:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760754.986137:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760754.986157:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760754.986267:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad880 - object 0x240000400:5 offset 25812992<>25817088 00000100:00100000:2.0:1789760754.986568:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760754.986588:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512668416:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760755.038419:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973621 gen 3 00000008:00010000:2.0:1789760755.038472:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760755.038480:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.038492:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.038522:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760755.038552:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512668416:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760755.097931:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6303, 6303] 00000080:00200000:3.0:1789760755.097945:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760755.097951:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760755.097966:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25821184 i_blocks 51207 00000020:00200000:3.0:1789760755.097969:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760755.097980:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760755.098014:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25821184, count: 4096 00000080:00200000:3.0:1789760755.098022:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25821184, bytes: 4096 00000080:00200000:3.0:1789760755.098032:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760755.098041:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760755.098043:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25821184, 25825280) 0 0 00000080:00200000:3.0:1789760755.098047:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760755.098065:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6304, 6304] 00020000:00200000:3.0:1789760755.098073:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25821184 chunk: [25821184, 25825280) 25825280, 4096 00020000:00200000:3.0:1789760755.098077:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760755.098096:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6304, 6304] 00000008:00010000:3.0:1789760755.098114:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760755.098120:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760755.098132:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25821184, 25825280) append 0 append lockpos 0 00000080:00200000:3.0:1789760755.098145:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25821184=0x18a0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760755.098203:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760755.098206:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25825280 00000020:00010000:3.0:1789760755.098214:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760755.098218:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.098221:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.098226:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760755.098229:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760755.098237:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760755.098245:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760755.098676:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760755.098791:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 25821184<>25825280 00000100:00100000:2.0:1789760755.098968:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512668672:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760755.134405:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973623 gen 3 00000008:00010000:2.0:1789760755.134458:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760755.134466:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.134479:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.134499:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760755.134529:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512668672:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760755.135082:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6304, 6304] 00000080:00200000:3.0:1789760755.135100:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760755.135106:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760755.135122:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25825280 i_blocks 51207 00000020:00200000:3.0:1789760755.135126:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760755.135135:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760755.135171:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25825280, count: 4096 00000080:00200000:3.0:1789760755.135179:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25825280, bytes: 4096 00000080:00200000:3.0:1789760755.135191:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760755.135201:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760755.135203:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25825280, 25829376) 0 0 00000080:00200000:3.0:1789760755.135208:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760755.135227:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6305, 6305] 00020000:00200000:3.0:1789760755.135236:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25825280 chunk: [25825280, 25829376) 25829376, 4096 00020000:00200000:3.0:1789760755.135242:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760755.135263:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6305, 6305] 00000008:00010000:3.0:1789760755.135282:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760755.135700:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760755.135717:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25825280, 25829376) append 0 append lockpos 0 00000080:00200000:3.0:1789760755.135732:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25825280=0x18a1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760755.135970:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760755.135974:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25829376 00000020:00010000:3.0:1789760755.135984:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760755.135988:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.136055:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.136062:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760755.136065:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760755.136075:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760755.136082:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760755.136106:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760755.136209:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae680 - object 0x240000400:5 offset 25825280<>25829376 00000100:00100000:2.0:1789760755.136302:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512668800:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760755.183335:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973624 gen 3 00000008:00010000:2.0:1789760755.183389:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760755.183398:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.183411:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.183495:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760755.183526:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512668800:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760755.187137:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760755.187179:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512668928:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760755.247450:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973625 gen 3 00000008:00010000:1.0:1789760755.247503:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760755.247512:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760755.247525:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760755.247551:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760755.247581:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512668928:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760755.248366:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512669056:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760755.336145:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973626 gen 3 00000008:00010000:1.0:1789760755.336198:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760755.336207:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760755.336218:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760755.336247:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760755.336279:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512669056:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760755.336565:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6307, 6307] 00000080:00200000:2.0:1789760755.336582:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760755.336589:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760755.336605:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25837568 i_blocks 51207 00000020:00200000:2.0:1789760755.336610:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760755.336620:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760755.336670:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25837568, count: 4096 00000080:00200000:2.0:1789760755.336678:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25837568, bytes: 4096 00000080:00200000:2.0:1789760755.336691:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760755.336702:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760755.336705:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25837568, 25841664) 0 0 00000080:00200000:2.0:1789760755.336716:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760755.336736:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6308, 6308] 00020000:00200000:2.0:1789760755.336745:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25837568 chunk: [25837568, 25841664) 25841664, 4096 00020000:00200000:2.0:1789760755.336750:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760755.336778:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6308, 6308] 00000008:00010000:2.0:1789760755.336797:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760755.336804:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760755.336816:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25837568, 25841664) append 0 append lockpos 0 00000080:00200000:2.0:1789760755.336830:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25837568=0x18a4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760755.336896:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760755.336947:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25841664 00000020:00010000:2.0:1789760755.336956:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760755.336960:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.336964:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.336968:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760755.336971:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760755.336979:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760755.336986:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760755.337003:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000008:00100000:2.0:1789760755.337108:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 25837568<>25841664 00000100:00100000:2.0:1789760755.337154:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760755.337171:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512669184:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760755.410544:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973627 gen 3 00000008:00010000:2.0:1789760755.410604:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:2.0:1789760755.410613:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.410624:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.410714:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:2.0:1789760755.410749:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512669184:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760755.411656:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512669312:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760755.439755:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973628 gen 3 00000008:00010000:1.0:1789760755.439945:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760755.439953:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760755.439965:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760755.440044:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760755.440075:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512669312:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760755.441889:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6309, 6309] 00000080:00200000:2.0:1789760755.441930:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760755.441938:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760755.441955:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25845760 i_blocks 51207 00000020:00200000:2.0:1789760755.441961:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760755.441978:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760755.442011:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25845760, count: 4096 00000080:00200000:2.0:1789760755.442020:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25845760, bytes: 4096 00000080:00200000:2.0:1789760755.442035:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760755.442044:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760755.442047:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25845760, 25849856) 0 0 00000080:00200000:2.0:1789760755.442052:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760755.442068:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6310, 6310] 00020000:00200000:2.0:1789760755.442078:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25845760 chunk: [25845760, 25849856) 25849856, 4096 00020000:00200000:2.0:1789760755.442082:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760755.442104:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6310, 6310] 00000008:00010000:2.0:1789760755.442124:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760755.442131:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760755.442144:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25845760, 25849856) append 0 append lockpos 0 00000080:00200000:2.0:1789760755.442159:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25845760=0x18a6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760755.442232:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760755.442235:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25849856 00000020:00010000:2.0:1789760755.442244:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760755.442247:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.442251:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.442254:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760755.442257:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760755.442266:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760755.442273:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760755.442304:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000008:00100000:2.0:1789760755.442426:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 25845760<>25849856 00000100:00100000:3.0:1789760755.442692:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760755.442729:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512669440:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760755.477996:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:2.0:1789760755.478023:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.478035:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.478058:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:2.0:1789760755.478090:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512669440:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760755.478359:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6310, 6310] 00000080:00200000:3.0:1789760755.478377:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760755.478383:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760755.478398:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25849856 i_blocks 51207 00000020:00200000:3.0:1789760755.478402:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760755.478412:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760755.478447:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25849856, count: 4096 00000080:00200000:3.0:1789760755.478455:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25849856, bytes: 4096 00000080:00200000:3.0:1789760755.478467:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760755.478477:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760755.478479:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25849856, 25853952) 0 0 00000080:00200000:3.0:1789760755.478484:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760755.478505:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6311, 6311] 00020000:00200000:3.0:1789760755.478514:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25849856 chunk: [25849856, 25853952) 25853952, 4096 00020000:00200000:3.0:1789760755.478519:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760755.478541:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6311, 6311] 00000008:00010000:3.0:1789760755.478561:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760755.478567:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760755.478579:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25849856, 25853952) append 0 append lockpos 0 00000080:00200000:3.0:1789760755.478594:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25849856=0x18a7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760755.478663:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760755.478666:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25853952 00000020:00010000:3.0:1789760755.478674:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760755.478679:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.478682:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.478686:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760755.478689:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760755.478696:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760755.478703:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760755.478723:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760755.480377:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 25849856<>25853952 00000100:00100000:3.0:1789760755.480459:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760755.480483:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512669568:192.168.203.156@tcp:4:dd.0 00010000:00010000:1.0:1789760755.486453:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760755.512773:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973630 gen 3 00000008:00010000:2.0:1789760755.512848:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760755.512857:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.512869:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.512891:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000080:00200000:3.0:1789760755.513313:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6311, 6311] 00000080:00200000:3.0:1789760755.513340:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760755.513346:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760755.513358:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25853952 i_blocks 51207 00000020:00200000:3.0:1789760755.513363:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760755.513372:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760755.513398:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25853952, count: 4096 00000080:00200000:3.0:1789760755.513405:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25853952, bytes: 4096 00000080:00200000:3.0:1789760755.513414:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760755.513422:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760755.513424:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25853952, 25858048) 0 0 00000080:00200000:3.0:1789760755.513427:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760755.513440:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6312, 6312] 00020000:00200000:3.0:1789760755.513447:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25853952 chunk: [25853952, 25858048) 25858048, 4096 00020000:00200000:3.0:1789760755.513450:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000100:00100000:2.0:1789760755.514109:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512669568:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760755.515625:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6312, 6312] 00000008:00010000:3.0:1789760755.515648:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760755.515653:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760755.515682:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25853952, 25858048) append 0 append lockpos 0 00000080:00200000:3.0:1789760755.515696:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25853952=0x18a8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760755.515983:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760755.515986:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25858048 00000020:00010000:3.0:1789760755.516056:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760755.516060:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.516063:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.516066:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760755.516069:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760755.516077:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760755.516085:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760755.516103:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760755.516268:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af800 - object 0x240000400:5 offset 25853952<>25858048 00000100:00100000:3.0:1789760755.516307:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760755.516333:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512669696:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760755.556101:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973631 gen 3 00000008:00010000:2.0:1789760755.556138:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760755.556146:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.556155:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.556175:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760755.556197:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512669696:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760755.556281:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6312, 6312] 00000080:00200000:3.0:1789760755.556293:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760755.556298:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760755.556376:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25858048 i_blocks 51207 00000020:00200000:3.0:1789760755.556381:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760755.556389:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760755.556412:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25858048, count: 4096 00000080:00200000:3.0:1789760755.556418:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25858048, bytes: 4096 00000080:00200000:3.0:1789760755.556425:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760755.556431:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760755.556434:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25858048, 25862144) 0 0 00000080:00200000:3.0:1789760755.556437:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760755.556449:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6313, 6313] 00020000:00200000:3.0:1789760755.556455:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25858048 chunk: [25858048, 25862144) 25862144, 4096 00020000:00200000:3.0:1789760755.556458:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760755.556472:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6313, 6313] 00000008:00010000:3.0:1789760755.556483:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760755.556488:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760755.556497:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25858048, 25862144) append 0 append lockpos 0 00000080:00200000:3.0:1789760755.556507:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25858048=0x18a9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760755.556546:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760755.556548:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25862144 00000020:00010000:3.0:1789760755.556554:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760755.556558:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.556561:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.556564:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760755.556566:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760755.556572:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760755.556576:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760755.556588:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760755.556665:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af800 - object 0x240000400:5 offset 25858048<>25862144 00000100:00100000:2.0:1789760755.556826:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512669824:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760755.608750:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973632 gen 3 00000008:00010000:2.0:1789760755.608787:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760755.608793:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.608803:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.608824:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760755.608848:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512669824:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760755.608871:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6313, 6313] 00000080:00200000:3.0:1789760755.608883:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760755.608888:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760755.608920:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25862144 i_blocks 51207 00000020:00200000:3.0:1789760755.608924:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760755.608931:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760755.608957:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25862144, count: 4096 00000080:00200000:3.0:1789760755.608962:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25862144, bytes: 4096 00000080:00200000:3.0:1789760755.608972:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760755.608978:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760755.608980:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25862144, 25866240) 0 0 00000080:00200000:3.0:1789760755.608983:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760755.609005:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6314, 6314] 00020000:00200000:3.0:1789760755.609011:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25862144 chunk: [25862144, 25866240) 25866240, 4096 00020000:00200000:3.0:1789760755.609015:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760755.609031:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6314, 6314] 00000008:00010000:3.0:1789760755.609110:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760755.609116:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760755.609125:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25862144, 25866240) append 0 append lockpos 0 00000080:00200000:3.0:1789760755.609137:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25862144=0x18aa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760755.609181:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760755.609183:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25866240 00000020:00010000:3.0:1789760755.609190:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760755.609193:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.609196:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.609199:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760755.609201:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760755.609207:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760755.609212:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760755.609227:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760755.609300:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 25862144<>25866240 00000100:00100000:2.0:1789760755.609426:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760755.609440:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512669952:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760755.655701:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973633 gen 3 00000008:00010000:2.0:1789760755.655752:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760755.655758:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.655768:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.655833:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760755.655856:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512669952:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760755.656190:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6314, 6314] 00000080:00200000:3.0:1789760755.656203:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760755.656207:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760755.656218:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25866240 i_blocks 51207 00000020:00200000:3.0:1789760755.656222:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760755.656229:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760755.656255:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25866240, count: 4096 00000080:00200000:3.0:1789760755.656261:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25866240, bytes: 4096 00000080:00200000:3.0:1789760755.656269:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760755.656276:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760755.656278:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25866240, 25870336) 0 0 00000080:00200000:3.0:1789760755.656282:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760755.656295:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6315, 6315] 00020000:00200000:3.0:1789760755.656302:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25866240 chunk: [25866240, 25870336) 25870336, 4096 00020000:00200000:3.0:1789760755.656306:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760755.656330:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6315, 6315] 00000008:00010000:3.0:1789760755.656343:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760755.656348:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760755.656357:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25866240, 25870336) append 0 append lockpos 0 00000080:00200000:3.0:1789760755.656367:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25866240=0x18ab000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760755.656412:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760755.656415:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25870336 00000020:00010000:3.0:1789760755.656421:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760755.656424:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.656427:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.656431:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760755.656433:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760755.656439:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760755.656444:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760755.656460:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760755.656539:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae680 - object 0x240000400:5 offset 25866240<>25870336 00000100:00100000:2.0:1789760755.656624:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760755.656637:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512670080:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760755.691212:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973634 gen 3 00000008:00010000:2.0:1789760755.691245:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760755.691251:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.691260:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.691337:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760755.691361:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512670080:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760755.694410:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6315, 6315] 00000080:00200000:3.0:1789760755.694431:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760755.694437:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760755.694451:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25870336 i_blocks 51207 00000020:00200000:3.0:1789760755.694456:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760755.694469:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760755.694496:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25870336, count: 4096 00000080:00200000:3.0:1789760755.694502:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25870336, bytes: 4096 00000080:00200000:3.0:1789760755.694513:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760755.694520:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760755.694523:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25870336, 25874432) 0 0 00000080:00200000:3.0:1789760755.694526:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760755.694539:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6316, 6316] 00020000:00200000:3.0:1789760755.694546:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25870336 chunk: [25870336, 25874432) 25874432, 4096 00020000:00200000:3.0:1789760755.694550:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760755.694565:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6316, 6316] 00000008:00010000:3.0:1789760755.694579:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760755.694584:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760755.694593:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25870336, 25874432) append 0 append lockpos 0 00000080:00200000:3.0:1789760755.694603:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25870336=0x18ac000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760755.694657:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760755.694659:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25874432 00000020:00010000:3.0:1789760755.694665:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760755.694669:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.694671:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.694674:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760755.694677:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760755.694683:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760755.694687:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760755.694706:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760755.694771:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 25870336<>25874432 00000100:00100000:2.0:1789760755.694895:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760755.694925:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512670208:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760755.735070:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760755.735082:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760755.735092:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760755.735182:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760755.735206:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512670208:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760755.735287:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6316, 6316] 00000080:00200000:3.0:1789760755.735298:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760755.735303:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760755.735315:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25874432 i_blocks 51207 00000020:00200000:3.0:1789760755.735327:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760755.735335:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760755.735360:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25874432, count: 4096 00000080:00200000:3.0:1789760755.735367:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25874432, bytes: 4096 00000080:00200000:3.0:1789760755.735377:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760755.735385:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760755.735387:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25874432, 25878528) 0 0 00000080:00200000:3.0:1789760755.735391:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760755.735404:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6317, 6317] 00020000:00200000:3.0:1789760755.735411:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25874432 chunk: [25874432, 25878528) 25878528, 4096 00020000:00200000:3.0:1789760755.735414:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760755.735428:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6317, 6317] 00000008:00010000:3.0:1789760755.735441:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760755.735446:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760755.735454:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25874432, 25878528) append 0 append lockpos 0 00000080:00200000:3.0:1789760755.735586:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25874432=0x18ad000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760755.735633:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760755.735636:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25878528 00000020:00010000:3.0:1789760755.735643:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760755.735647:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.735649:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760755.735653:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760755.735655:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760755.735661:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760755.735667:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760755.735680:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760755.735831:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad880 - object 0x240000400:5 offset 25874432<>25878528 00000100:00100000:2.0:1789760755.735889:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512670336:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760755.807700:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973636 gen 3 00000008:00010000:3.0:1789760755.807739:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760755.807745:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760755.807752:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760755.807769:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760755.807786:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512670336:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760755.808022:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6317, 6317] 00000080:00200000:2.0:1789760755.808035:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760755.808040:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760755.808051:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25878528 i_blocks 51207 00000020:00200000:2.0:1789760755.808054:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760755.808061:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760755.808085:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25878528, count: 4096 00000080:00200000:2.0:1789760755.808090:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25878528, bytes: 4096 00000080:00200000:2.0:1789760755.808098:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760755.808105:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760755.808107:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25878528, 25882624) 0 0 00000080:00200000:2.0:1789760755.808110:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760755.808122:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6318, 6318] 00020000:00200000:2.0:1789760755.808127:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25878528 chunk: [25878528, 25882624) 25882624, 4096 00020000:00200000:2.0:1789760755.808130:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760755.808143:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6318, 6318] 00000008:00010000:2.0:1789760755.808158:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760755.808162:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760755.808171:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25878528, 25882624) append 0 append lockpos 0 00000080:00200000:2.0:1789760755.808181:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25878528=0x18ae000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760755.808219:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760755.808222:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25882624 00000020:00010000:2.0:1789760755.808228:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760755.808231:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.808233:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.808236:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760755.808239:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760755.808244:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760755.808249:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760755.808260:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000008:00100000:2.0:1789760755.808340:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 25878528<>25882624 00000100:00100000:3.0:1789760755.808418:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512670464:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760755.889367:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973637 gen 3 00000008:00010000:3.0:1789760755.889399:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760755.889405:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760755.889412:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760755.889430:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760755.889451:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512670464:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760755.889593:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6318, 6318] 00000080:00200000:2.0:1789760755.889605:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760755.889609:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760755.889622:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25882624 i_blocks 51207 00000020:00200000:2.0:1789760755.889625:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760755.889633:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760755.889658:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25882624, count: 4096 00000080:00200000:2.0:1789760755.889665:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25882624, bytes: 4096 00000080:00200000:2.0:1789760755.889672:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760755.889679:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760755.889681:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25882624, 25886720) 0 0 00000080:00200000:2.0:1789760755.889685:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760755.889697:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6319, 6319] 00020000:00200000:2.0:1789760755.889705:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25882624 chunk: [25882624, 25886720) 25886720, 4096 00020000:00200000:2.0:1789760755.889709:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760755.889723:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6319, 6319] 00000008:00010000:2.0:1789760755.889737:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760755.889742:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760755.889750:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25882624, 25886720) append 0 append lockpos 0 00000080:00200000:2.0:1789760755.889760:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25882624=0x18af000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760755.889803:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760755.889806:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25886720 00000020:00010000:2.0:1789760755.889812:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760755.889815:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.889818:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760755.889821:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760755.889823:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760755.889829:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760755.889835:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760755.889848:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760755.890055:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 25882624<>25886720 00000100:00100000:3.0:1789760755.890119:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512670592:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760755.930181:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760755.930192:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760755.930200:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760755.930223:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760755.930248:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512670592:192.168.203.156@tcp:4:dd.0 00020000:00200000:0.0:1789760755.930491:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25886720 chunk: [25886720, 25890816) 25890816, 4096 00020000:00200000:0.0:1789760755.930495:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760755.930509:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6320, 6320] 00000008:00010000:0.0:1789760755.930521:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760755.930525:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760755.930534:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25886720, 25890816) append 0 append lockpos 0 00000080:00200000:0.0:1789760755.930543:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25886720=0x18b0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760755.930583:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760755.930585:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25890816 00000020:00010000:0.0:1789760755.930591:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760755.930594:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760755.930597:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760755.930600:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760755.930602:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760755.930608:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760755.930613:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760755.930626:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc180 00000008:00100000:0.0:1789760755.930697:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b800 - object 0x240000400:5 offset 25886720<>25890816 00000100:00100000:1.0:1789760755.930850:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512670720:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760755.981052:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973638 gen 3 00000100:00100000:1.0:1789760755.981096:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00514b800 x1876699512670720/t4294973639(4294973639) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760771 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760755.981147:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760755.981208:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760755.981220:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760755.981242:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760755.981268:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512670720:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760755.981484:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6320, 6320] 00000080:00200000:0.0:1789760755.981559:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760755.981565:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760755.981579:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25890816 i_blocks 51207 00000020:00200000:0.0:1789760755.981584:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760755.981598:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760755.981623:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25890816, count: 4096 00000080:00200000:0.0:1789760755.981630:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25890816, bytes: 4096 00000080:00200000:0.0:1789760755.981642:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760755.981649:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760755.981651:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25890816, 25894912) 0 0 00000080:00200000:0.0:1789760755.981655:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760755.981668:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6321, 6321] 00020000:00200000:0.0:1789760755.981675:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25890816 chunk: [25890816, 25894912) 25894912, 4096 00020000:00200000:0.0:1789760755.981678:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760755.981695:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6321, 6321] 00000008:00010000:0.0:1789760755.981710:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760755.981715:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760755.981723:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25890816, 25894912) append 0 append lockpos 0 00000080:00200000:0.0:1789760755.981733:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25890816=0x18b1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760755.981785:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760755.981788:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25894912 00000020:00010000:0.0:1789760755.981795:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760755.981799:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760755.981801:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760755.981805:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760755.981807:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760755.981814:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760755.981819:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760755.981837:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccd80 00000008:00100000:0.0:1789760755.981968:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148380 - object 0x240000400:5 offset 25890816<>25894912 00000100:00100000:1.0:1789760755.982143:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760755.982164:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512670848:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760756.029370:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973640 gen 3 00000008:00010000:1.0:1789760756.029416:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760756.029423:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760756.029431:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760756.029512:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760756.029537:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512670848:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760756.029604:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6321, 6321] 00000080:00200000:0.0:1789760756.029617:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760756.029622:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760756.029634:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25894912 i_blocks 51207 00000020:00200000:0.0:1789760756.029637:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760756.029645:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760756.029670:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25894912, count: 4096 00000080:00200000:0.0:1789760756.029676:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25894912, bytes: 4096 00000080:00200000:0.0:1789760756.029683:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760756.029690:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760756.029692:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25894912, 25899008) 0 0 00000080:00200000:0.0:1789760756.029696:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760756.029774:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6322, 6322] 00020000:00200000:0.0:1789760756.029782:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25894912 chunk: [25894912, 25899008) 25899008, 4096 00020000:00200000:0.0:1789760756.029840:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760756.029860:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6322, 6322] 00000008:00010000:0.0:1789760756.029958:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760756.029963:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760756.029972:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25894912, 25899008) append 0 append lockpos 0 00000080:00200000:0.0:1789760756.030046:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25894912=0x18b2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760756.030090:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760756.030093:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25899008 00000020:00010000:0.0:1789760756.030099:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760756.030102:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.030105:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.030108:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760756.030111:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760756.030117:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760756.030122:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760756.030208:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc780 00000008:00100000:0.0:1789760756.030350:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b480 - object 0x240000400:5 offset 25894912<>25899008 00000100:00100000:1.0:1789760756.030406:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512670976:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760756.085536:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973641 gen 3 00000008:00010000:1.0:1789760756.085573:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760756.085579:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760756.085589:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760756.085609:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760756.085632:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512670976:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760756.086228:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6322, 6322] 00000080:00200000:0.0:1789760756.086242:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760756.086248:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760756.086259:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25899008 i_blocks 51207 00000020:00200000:0.0:1789760756.086262:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760756.086271:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760756.086297:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25899008, count: 4096 00000080:00200000:0.0:1789760756.086303:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25899008, bytes: 4096 00000080:00200000:0.0:1789760756.086311:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760756.086328:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760756.086330:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25899008, 25903104) 0 0 00000080:00200000:0.0:1789760756.086333:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760756.086347:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6323, 6323] 00020000:00200000:0.0:1789760756.086354:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25899008 chunk: [25899008, 25903104) 25903104, 4096 00020000:00200000:0.0:1789760756.086358:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760756.086371:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6323, 6323] 00000008:00010000:0.0:1789760756.086385:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760756.086389:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760756.086397:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25899008, 25903104) append 0 append lockpos 0 00000080:00200000:0.0:1789760756.086408:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25899008=0x18b3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760756.086453:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760756.086455:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25903104 00000020:00010000:0.0:1789760756.086461:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760756.086464:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.086467:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.086470:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760756.086472:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760756.086478:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760756.086484:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760756.086499:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000008:00100000:0.0:1789760756.086579:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149880 - object 0x240000400:5 offset 25899008<>25903104 00000100:00100000:1.0:1789760756.086684:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512671104:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760756.124263:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973642 gen 3 00000008:00010000:1.0:1789760756.124300:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760756.124308:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760756.124329:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760756.124348:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760756.124370:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512671104:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760756.124593:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6323, 6323] 00000080:00200000:0.0:1789760756.124606:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760756.124613:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760756.124626:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25903104 i_blocks 51207 00000020:00200000:0.0:1789760756.124630:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760756.124638:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760756.124664:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25903104, count: 4096 00000080:00200000:0.0:1789760756.124670:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25903104, bytes: 4096 00000080:00200000:0.0:1789760756.124678:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760756.124686:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760756.124687:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25903104, 25907200) 0 0 00000080:00200000:0.0:1789760756.124691:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760756.124772:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6324, 6324] 00020000:00200000:0.0:1789760756.124780:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25903104 chunk: [25903104, 25907200) 25907200, 4096 00020000:00200000:0.0:1789760756.124784:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760756.124853:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6324, 6324] 00000008:00010000:0.0:1789760756.125084:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760756.125091:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760756.125101:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25903104, 25907200) append 0 append lockpos 0 00000080:00200000:0.0:1789760756.125112:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25903104=0x18b4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760756.125241:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760756.125244:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25907200 00000020:00010000:0.0:1789760756.125251:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760756.125254:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.125257:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.125261:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760756.125264:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760756.125271:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760756.125277:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760756.125293:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc000 00000008:00100000:0.0:1789760756.125386:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a300 - object 0x240000400:5 offset 25903104<>25907200 00000100:00100000:1.0:1789760756.125455:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512671232:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760756.166773:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973643 gen 3 00000008:00010000:1.0:1789760756.166819:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760756.166825:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760756.166834:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760756.166855:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000080:00200000:0.0:1789760756.167212:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6324, 6324] 00000080:00200000:0.0:1789760756.167226:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760756.167231:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760756.167243:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25907200 i_blocks 51207 00000020:00200000:0.0:1789760756.167247:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760756.167254:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760756.167282:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25907200, count: 4096 00000080:00200000:0.0:1789760756.167288:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25907200, bytes: 4096 00000080:00200000:0.0:1789760756.167296:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760756.167302:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760756.167303:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25907200, 25911296) 0 0 00000080:00200000:0.0:1789760756.167307:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760756.167329:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6325, 6325] 00020000:00200000:0.0:1789760756.167339:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25907200 chunk: [25907200, 25911296) 25911296, 4096 00020000:00200000:0.0:1789760756.167344:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760756.167360:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6325, 6325] 00000008:00010000:0.0:1789760756.167372:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760756.167377:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760756.167386:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25907200, 25911296) append 0 append lockpos 0 00000080:00200000:0.0:1789760756.167395:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25907200=0x18b5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760756.167433:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760756.167436:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25911296 00000020:00010000:0.0:1789760756.167442:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760756.167445:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.167449:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.167452:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760756.167454:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760756.167461:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760756.167467:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1789760756.167513:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512671232:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760756.167549:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccf00 00000008:00100000:1.0:1789760756.167630:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 25907200<>25911296 00000100:00100000:1.0:1789760756.167740:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760756.167751:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512671360:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760756.235050:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973644 gen 3 00000008:00010000:1.0:1789760756.235086:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760756.235092:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760756.235102:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760756.235123:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760756.235147:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512671360:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760756.235782:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6325, 6325] 00000080:00200000:0.0:1789760756.235807:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760756.235812:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760756.235824:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25911296 i_blocks 51207 00000020:00200000:0.0:1789760756.235827:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760756.235836:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760756.235860:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25911296, count: 4096 00000080:00200000:0.0:1789760756.235866:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25911296, bytes: 4096 00000080:00200000:0.0:1789760756.235874:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760756.235881:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760756.235883:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25911296, 25915392) 0 0 00000080:00200000:0.0:1789760756.235886:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760756.235924:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6326, 6326] 00020000:00200000:0.0:1789760756.235932:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25911296 chunk: [25911296, 25915392) 25915392, 4096 00020000:00200000:0.0:1789760756.235935:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760756.235950:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6326, 6326] 00000008:00010000:0.0:1789760756.235964:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760756.235969:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760756.235978:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25911296, 25915392) append 0 append lockpos 0 00000080:00200000:0.0:1789760756.235988:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25911296=0x18b6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760756.236031:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760756.236034:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25915392 00000020:00010000:0.0:1789760756.236041:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760756.236044:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.236047:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760756.236050:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760756.236053:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760756.236059:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760756.236064:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760756.236280:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cca80 00000008:00100000:0.0:1789760756.236374:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a680 - object 0x240000400:5 offset 25911296<>25915392 00000100:00100000:0.0:1789760756.236415:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760756.236431:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512671488:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760756.302378:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:0.0:1789760756.302389:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760756.302399:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760756.302472:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:0.0:1789760756.302496:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512671488:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760756.303610:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6326, 6326] 00000080:00200000:3.0:1789760756.303625:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760756.303630:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760756.303642:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25915392 i_blocks 51207 00000020:00200000:3.0:1789760756.303645:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760756.303656:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760756.303679:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25915392, count: 4096 00000080:00200000:3.0:1789760756.303685:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25915392, bytes: 4096 00000080:00200000:3.0:1789760756.303696:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760756.303703:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760756.303705:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25915392, 25919488) 0 0 00000080:00200000:3.0:1789760756.303710:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760756.303725:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6327, 6327] 00020000:00200000:3.0:1789760756.303733:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25915392 chunk: [25915392, 25919488) 25919488, 4096 00020000:00200000:3.0:1789760756.303736:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760756.303754:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6327, 6327] 00000008:00010000:3.0:1789760756.303770:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760756.303775:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760756.303785:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25915392, 25919488) append 0 append lockpos 0 00000080:00200000:3.0:1789760756.303803:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25915392=0x18b7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760756.303873:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760756.303876:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25919488 00000020:00010000:3.0:1789760756.303893:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760756.303933:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760756.303936:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760756.303940:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760756.303942:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760756.303949:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760756.303956:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760756.306086:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760756.306186:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad880 - object 0x240000400:5 offset 25915392<>25919488 00000100:00100000:3.0:1789760756.306397:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760756.306416:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512671616:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760756.379994:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973645 gen 3 00000100:00100000:3.0:1789760756.380028:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc0107ad880 x1876699512671616/t4294973646(4294973646) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760772 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760756.380067:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760756.380072:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760756.380081:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760756.380102:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760756.380121:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512671616:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760756.380252:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6327, 6327] 00000080:00200000:2.0:1789760756.380264:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760756.380269:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760756.380281:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25919488 i_blocks 51207 00000020:00200000:2.0:1789760756.380284:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760756.380293:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760756.380332:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25919488, count: 4096 00000080:00200000:2.0:1789760756.380338:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25919488, bytes: 4096 00000080:00200000:2.0:1789760756.380346:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760756.380352:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760756.380355:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25919488, 25923584) 0 0 00000080:00200000:2.0:1789760756.380358:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760756.380369:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6328, 6328] 00020000:00200000:2.0:1789760756.380375:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25919488 chunk: [25919488, 25923584) 25923584, 4096 00020000:00200000:2.0:1789760756.380379:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760756.380393:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6328, 6328] 00000008:00010000:2.0:1789760756.380405:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760756.380410:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760756.380420:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25919488, 25923584) append 0 append lockpos 0 00000080:00200000:2.0:1789760756.380430:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25919488=0x18b8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760756.380474:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760756.380476:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25923584 00000020:00010000:2.0:1789760756.380482:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760756.380485:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.380488:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.380491:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760756.380494:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760756.380500:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760756.380505:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760756.380519:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760756.380593:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 25919488<>25923584 00000100:00100000:2.0:1789760756.380629:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760756.380640:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512671744:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760756.418413:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973646 gen 3 00000100:00100000:3.0:1789760756.418440:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368f800 x1876699512671744/t4294973647(4294973647) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760772 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760756.418477:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760756.418483:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760756.418491:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760756.418548:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760756.418569:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512671744:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760756.418695:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6328, 6328] 00000080:00200000:2.0:1789760756.418757:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760756.418765:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760756.418776:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25923584 i_blocks 51207 00000020:00200000:2.0:1789760756.418780:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760756.418788:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760756.418815:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25923584, count: 4096 00000080:00200000:2.0:1789760756.418821:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25923584, bytes: 4096 00000080:00200000:2.0:1789760756.418828:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760756.418835:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760756.418837:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25923584, 25927680) 0 0 00000080:00200000:2.0:1789760756.418840:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760756.418861:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6329, 6329] 00020000:00200000:2.0:1789760756.419027:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25923584 chunk: [25923584, 25927680) 25927680, 4096 00020000:00200000:2.0:1789760756.419040:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760756.419058:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6329, 6329] 00000008:00010000:2.0:1789760756.419072:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760756.419076:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760756.419085:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25923584, 25927680) append 0 append lockpos 0 00000080:00200000:2.0:1789760756.419096:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25923584=0x18b9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760756.419154:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760756.419157:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25927680 00000020:00010000:2.0:1789760756.419164:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760756.419167:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.419170:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.419173:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760756.419176:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760756.419182:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760756.419188:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760756.419202:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760756.419289:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 25923584<>25927680 00000100:00100000:3.0:1789760756.419816:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512671872:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760756.457215:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973648 gen 3 00000008:00010000:3.0:1789760756.457264:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760756.457271:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760756.457279:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760756.457298:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760756.457334:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512671872:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760756.459187:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6329, 6329] 00000080:00200000:2.0:1789760756.459205:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760756.459212:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760756.459225:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25927680 i_blocks 51207 00000020:00200000:2.0:1789760756.459229:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760756.459240:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760756.459265:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25927680, count: 4096 00000080:00200000:2.0:1789760756.459272:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25927680, bytes: 4096 00000080:00200000:2.0:1789760756.459282:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760756.459289:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760756.459291:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25927680, 25931776) 0 0 00000080:00200000:2.0:1789760756.459295:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760756.459308:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6330, 6330] 00020000:00200000:2.0:1789760756.459326:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25927680 chunk: [25927680, 25931776) 25931776, 4096 00020000:00200000:2.0:1789760756.459330:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760756.459347:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6330, 6330] 00000008:00010000:2.0:1789760756.459361:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760756.459366:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760756.459375:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25927680, 25931776) append 0 append lockpos 0 00000080:00200000:2.0:1789760756.459386:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25927680=0x18ba000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760756.459561:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760756.459564:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25931776 00000020:00010000:2.0:1789760756.459572:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760756.459576:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.459579:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.459582:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760756.459585:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760756.459592:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760756.459598:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760756.459615:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760756.459692:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 25927680<>25931776 00000100:00100000:3.0:1789760756.460124:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760756.460144:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512672000:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760756.509245:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760756.537831:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973649 gen 3 00000008:00010000:3.0:1789760756.537866:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760756.537873:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760756.537881:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760756.538000:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760756.538024:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512672000:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760756.538093:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6330, 6330] 00000080:00200000:2.0:1789760756.538105:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760756.538110:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760756.538120:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25931776 i_blocks 51207 00000020:00200000:2.0:1789760756.538123:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760756.538131:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760756.538156:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25931776, count: 4096 00000080:00200000:2.0:1789760756.538163:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25931776, bytes: 4096 00000080:00200000:2.0:1789760756.538170:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760756.538176:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760756.538178:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25931776, 25935872) 0 0 00000080:00200000:2.0:1789760756.538182:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760756.538261:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6331, 6331] 00020000:00200000:2.0:1789760756.538269:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25931776 chunk: [25931776, 25935872) 25935872, 4096 00020000:00200000:2.0:1789760756.538272:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760756.538287:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6331, 6331] 00000008:00010000:2.0:1789760756.538300:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760756.538305:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760756.538327:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25931776, 25935872) append 0 append lockpos 0 00000080:00200000:2.0:1789760756.538339:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25931776=0x18bb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760756.538381:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760756.538383:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25935872 00000020:00010000:2.0:1789760756.538390:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760756.538393:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.538396:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.538399:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760756.538401:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760756.538408:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760756.538413:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760756.538426:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760756.538502:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 25931776<>25935872 00000100:00100000:3.0:1789760756.538780:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512672128:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760756.616634:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973650 gen 3 00000008:00010000:3.0:1789760756.616672:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760756.616677:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760756.616686:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760756.616711:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760756.616734:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512672128:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760756.617156:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6331, 6331] 00000080:00200000:2.0:1789760756.617170:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760756.617174:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760756.617185:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25935872 i_blocks 51207 00000020:00200000:2.0:1789760756.617188:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760756.617196:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760756.617220:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25935872, count: 4096 00000080:00200000:2.0:1789760756.617226:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25935872, bytes: 4096 00000080:00200000:2.0:1789760756.617237:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760756.617243:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760756.617245:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25935872, 25939968) 0 0 00000080:00200000:2.0:1789760756.617249:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760756.617261:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6332, 6332] 00020000:00200000:2.0:1789760756.617267:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25935872 chunk: [25935872, 25939968) 25939968, 4096 00020000:00200000:2.0:1789760756.617271:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760756.617284:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6332, 6332] 00000008:00010000:2.0:1789760756.617298:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760756.617302:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760756.617326:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25935872, 25939968) append 0 append lockpos 0 00000080:00200000:2.0:1789760756.617337:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25935872=0x18bc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760756.617380:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760756.617382:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25939968 00000020:00010000:2.0:1789760756.617388:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760756.617391:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.617394:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.617397:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760756.617400:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760756.617407:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760756.617412:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760756.617426:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760756.617607:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 25935872<>25939968 00000100:00100000:3.0:1789760756.617763:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512672256:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760756.663654:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973651 gen 3 00000008:00010000:3.0:1789760756.663690:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760756.663697:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760756.663714:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760756.663730:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760756.663753:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512672256:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760756.664005:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6332, 6332] 00000080:00200000:2.0:1789760756.664018:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760756.664023:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760756.664034:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25939968 i_blocks 51207 00000020:00200000:2.0:1789760756.664037:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760756.664158:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760756.664183:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25939968, count: 4096 00000080:00200000:2.0:1789760756.664190:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25939968, bytes: 4096 00000080:00200000:2.0:1789760756.664198:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760756.664205:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760756.664207:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25939968, 25944064) 0 0 00000080:00200000:2.0:1789760756.664211:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760756.664225:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6333, 6333] 00020000:00200000:2.0:1789760756.664231:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25939968 chunk: [25939968, 25944064) 25944064, 4096 00020000:00200000:2.0:1789760756.664235:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760756.664250:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6333, 6333] 00000008:00010000:2.0:1789760756.664265:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760756.664269:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760756.664278:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25939968, 25944064) append 0 append lockpos 0 00000080:00200000:2.0:1789760756.664301:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25939968=0x18bd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760756.664356:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760756.664358:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25944064 00000020:00010000:2.0:1789760756.664365:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760756.664368:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.664372:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.664375:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760756.664378:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760756.664384:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760756.664390:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760756.664403:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760756.664646:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 25939968<>25944064 00000100:00100000:3.0:1789760756.664936:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512672384:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760756.714675:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973652 gen 3 00000008:00010000:3.0:1789760756.714722:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760756.714728:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760756.714736:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760756.714751:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760756.714777:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512672384:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760756.715096:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6333, 6333] 00000080:00200000:2.0:1789760756.715110:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760756.715115:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760756.715128:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25944064 i_blocks 51207 00000020:00200000:2.0:1789760756.715131:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760756.715139:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760756.715163:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25944064, count: 4096 00000080:00200000:2.0:1789760756.715169:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25944064, bytes: 4096 00000080:00200000:2.0:1789760756.715176:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760756.715183:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760756.715186:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25944064, 25948160) 0 0 00000080:00200000:2.0:1789760756.715189:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760756.715202:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6334, 6334] 00020000:00200000:2.0:1789760756.715209:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25944064 chunk: [25944064, 25948160) 25948160, 4096 00020000:00200000:2.0:1789760756.715212:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760756.715227:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6334, 6334] 00000008:00010000:2.0:1789760756.715240:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760756.715245:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760756.715254:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25944064, 25948160) append 0 append lockpos 0 00000080:00200000:2.0:1789760756.715265:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25944064=0x18be000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760756.715307:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760756.715326:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25948160 00000020:00010000:2.0:1789760756.715333:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760756.715335:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.715338:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.715341:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760756.715343:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760756.715350:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760756.715356:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760756.715370:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760756.715443:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 25944064<>25948160 00000100:00100000:3.0:1789760756.715596:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512672512:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760756.765755:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973653 gen 3 00000008:00010000:3.0:1789760756.765994:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760756.766001:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760756.766011:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760756.766028:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760756.766050:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512672512:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760756.766141:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6334, 6334] 00000080:00200000:2.0:1789760756.766154:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760756.766159:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760756.766171:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25948160 i_blocks 51207 00000020:00200000:2.0:1789760756.766174:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760756.766182:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760756.766207:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25948160, count: 4096 00000080:00200000:2.0:1789760756.766213:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25948160, bytes: 4096 00000080:00200000:2.0:1789760756.766220:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760756.766227:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760756.766229:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25948160, 25952256) 0 0 00000080:00200000:2.0:1789760756.766233:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760756.766246:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6335, 6335] 00020000:00200000:2.0:1789760756.766253:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25948160 chunk: [25948160, 25952256) 25952256, 4096 00020000:00200000:2.0:1789760756.766256:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760756.766270:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6335, 6335] 00000008:00010000:2.0:1789760756.766285:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760756.766290:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760756.766300:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25948160, 25952256) append 0 append lockpos 0 00000080:00200000:2.0:1789760756.766323:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25948160=0x18bf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760756.766366:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760756.766369:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25952256 00000020:00010000:2.0:1789760756.766375:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760756.766378:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.766381:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.766384:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760756.766387:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760756.766392:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760756.766398:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760756.766412:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760756.766484:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 25948160<>25952256 00000100:00100000:3.0:1789760756.766539:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512672640:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760756.803796:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973654 gen 3 00000008:00010000:3.0:1789760756.803831:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760756.803838:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760756.803847:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760756.803864:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760756.803887:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512672640:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760756.804093:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6335, 6335] 00000080:00200000:2.0:1789760756.804106:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760756.804110:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760756.804122:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25952256 i_blocks 51207 00000020:00200000:2.0:1789760756.804126:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760756.804133:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760756.804166:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25952256, count: 4096 00000080:00200000:2.0:1789760756.804172:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25952256, bytes: 4096 00000080:00200000:2.0:1789760756.804180:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760756.804186:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760756.804188:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25952256, 25956352) 0 0 00000080:00200000:2.0:1789760756.804192:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760756.804205:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6336, 6336] 00020000:00200000:2.0:1789760756.804212:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25952256 chunk: [25952256, 25956352) 25956352, 4096 00020000:00200000:2.0:1789760756.804215:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760756.804231:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6336, 6336] 00000008:00010000:2.0:1789760756.804245:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760756.804250:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760756.804259:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25952256, 25956352) append 0 append lockpos 0 00000080:00200000:2.0:1789760756.804270:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25952256=0x18c0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760756.804325:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760756.804328:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25956352 00000020:00010000:2.0:1789760756.804334:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760756.804336:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.804339:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.804341:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760756.804344:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760756.804350:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760756.804355:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760756.804369:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760756.804438:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 25952256<>25956352 00000100:00100000:3.0:1789760756.804600:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512672768:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760756.849828:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760756.849839:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760756.849848:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760756.849865:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760756.849891:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512672768:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760756.850112:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6336, 6336] 00000080:00200000:2.0:1789760756.850124:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760756.850129:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760756.850217:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25956352 i_blocks 51207 00000020:00200000:2.0:1789760756.850221:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760756.850228:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760756.850254:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25956352, count: 4096 00000080:00200000:2.0:1789760756.850259:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25956352, bytes: 4096 00000080:00200000:2.0:1789760756.850267:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760756.850274:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760756.850275:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25956352, 25960448) 0 0 00000080:00200000:2.0:1789760756.850279:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760756.850292:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6337, 6337] 00020000:00200000:2.0:1789760756.850299:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25956352 chunk: [25956352, 25960448) 25960448, 4096 00020000:00200000:2.0:1789760756.850302:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760756.850330:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6337, 6337] 00000008:00010000:2.0:1789760756.850344:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760756.850348:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760756.850357:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25956352, 25960448) append 0 append lockpos 0 00000080:00200000:2.0:1789760756.850368:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25956352=0x18c1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760756.850412:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760756.850415:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25960448 00000020:00010000:2.0:1789760756.850421:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760756.850424:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.850427:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.850430:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760756.850433:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760756.850439:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760756.850445:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760756.850459:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760756.850536:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 25956352<>25960448 00000100:00100000:3.0:1789760756.850764:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512672896:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760756.898784:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973655 gen 3 00000100:00100000:3.0:1789760756.898815:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368d180 x1876699512672896/t4294973656(4294973656) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760772 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760756.899144:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760756.899151:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760756.899160:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760756.899225:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760756.899247:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512672896:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760756.899377:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6337, 6337] 00000080:00200000:2.0:1789760756.899389:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760756.899394:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760756.899540:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25960448 i_blocks 51207 00000020:00200000:2.0:1789760756.899544:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760756.899552:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760756.899577:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25960448, count: 4096 00000080:00200000:2.0:1789760756.900110:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25960448, bytes: 4096 00000080:00200000:2.0:1789760756.900122:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760756.900130:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760756.900132:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25960448, 25964544) 0 0 00000080:00200000:2.0:1789760756.900135:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760756.900149:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6338, 6338] 00020000:00200000:2.0:1789760756.900156:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25960448 chunk: [25960448, 25964544) 25964544, 4096 00020000:00200000:2.0:1789760756.900160:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760756.900175:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6338, 6338] 00000008:00010000:2.0:1789760756.900188:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760756.900192:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760756.900201:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25960448, 25964544) append 0 append lockpos 0 00000080:00200000:2.0:1789760756.900212:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25960448=0x18c2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760756.900255:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760756.900257:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25964544 00000020:00010000:2.0:1789760756.900264:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760756.900267:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.900270:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.900273:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760756.900276:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760756.900282:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760756.900287:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760756.900303:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760756.900685:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 25960448<>25964544 00000100:00100000:3.0:1789760756.900804:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512673024:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760756.947039:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973657 gen 3 00000008:00010000:3.0:1789760756.947089:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760756.947096:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760756.947103:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760756.947121:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760756.947145:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512673024:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760756.947281:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6338, 6338] 00000080:00200000:2.0:1789760756.947295:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760756.947300:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760756.947326:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25964544 i_blocks 51207 00000020:00200000:2.0:1789760756.947330:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760756.947337:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760756.947362:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25964544, count: 4096 00000080:00200000:2.0:1789760756.947368:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25964544, bytes: 4096 00000080:00200000:2.0:1789760756.947375:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760756.947383:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760756.947385:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25964544, 25968640) 0 0 00000080:00200000:2.0:1789760756.947389:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760756.947400:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6339, 6339] 00020000:00200000:2.0:1789760756.947407:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25964544 chunk: [25964544, 25968640) 25968640, 4096 00020000:00200000:2.0:1789760756.947410:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760756.947426:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6339, 6339] 00000008:00010000:2.0:1789760756.947502:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760756.947507:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760756.947517:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25964544, 25968640) append 0 append lockpos 0 00000080:00200000:2.0:1789760756.947528:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25964544=0x18c3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760756.947571:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760756.947573:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25968640 00000020:00010000:2.0:1789760756.947580:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760756.947583:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.947586:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.947589:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760756.947591:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760756.947597:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760756.947602:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760756.947616:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760756.947687:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 25964544<>25968640 00000100:00100000:3.0:1789760756.947892:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512673152:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760756.983256:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973658 gen 3 00000008:00010000:3.0:1789760756.983291:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760756.983298:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760756.983314:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760756.983331:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760756.983354:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512673152:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760756.983625:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6339, 6339] 00000080:00200000:2.0:1789760756.983637:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760756.983642:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760756.983654:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25968640 i_blocks 51207 00000020:00200000:2.0:1789760756.983657:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760756.983664:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760756.983688:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25968640, count: 4096 00000080:00200000:2.0:1789760756.983695:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25968640, bytes: 4096 00000080:00200000:2.0:1789760756.983702:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760756.983709:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760756.983712:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25968640, 25972736) 0 0 00000080:00200000:2.0:1789760756.983715:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760756.983728:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6340, 6340] 00020000:00200000:2.0:1789760756.983734:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25968640 chunk: [25968640, 25972736) 25972736, 4096 00020000:00200000:2.0:1789760756.983738:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760756.984158:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6340, 6340] 00000008:00010000:2.0:1789760756.984173:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760756.984178:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760756.984188:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25968640, 25972736) append 0 append lockpos 0 00000080:00200000:2.0:1789760756.984199:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25968640=0x18c4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760756.984245:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760756.984248:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25972736 00000020:00010000:2.0:1789760756.984254:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760756.984257:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.984260:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760756.984264:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760756.984266:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760756.984273:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760756.984278:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760756.984292:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760756.984377:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 25968640<>25972736 00000100:00100000:3.0:1789760756.984491:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760756.984507:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512673280:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760757.034260:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973659 gen 3 00000008:00010000:3.0:1789760757.034296:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760757.034302:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760757.034318:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760757.034373:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760757.034398:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512673280:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760757.034467:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6340, 6340] 00000080:00200000:2.0:1789760757.034479:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760757.034485:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760757.034497:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25972736 i_blocks 51207 00000020:00200000:2.0:1789760757.034500:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760757.034508:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760757.034534:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25972736, count: 4096 00000080:00200000:2.0:1789760757.034541:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25972736, bytes: 4096 00000080:00200000:2.0:1789760757.034548:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760757.034555:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760757.034557:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25972736, 25976832) 0 0 00000080:00200000:2.0:1789760757.034561:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760757.034574:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6341, 6341] 00020000:00200000:2.0:1789760757.034581:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25972736 chunk: [25972736, 25976832) 25976832, 4096 00020000:00200000:2.0:1789760757.034585:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760757.034601:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6341, 6341] 00000008:00010000:2.0:1789760757.034614:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760757.034619:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760757.034628:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25972736, 25976832) append 0 append lockpos 0 00000080:00200000:2.0:1789760757.034641:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25972736=0x18c5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760757.034686:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760757.034688:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25976832 00000020:00010000:2.0:1789760757.034694:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760757.034697:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760757.034700:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760757.034703:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760757.034706:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760757.034713:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760757.034718:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760757.034732:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760757.034806:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 25972736<>25976832 00000100:00100000:3.0:1789760757.034859:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512673408:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760757.068292:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973660 gen 3 00000008:00010000:3.0:1789760757.068341:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760757.068348:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760757.068356:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760757.068464:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000080:00200000:2.0:1789760757.068487:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6341, 6341] 00000100:00100000:3.0:1789760757.068494:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512673408:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760757.068501:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760757.068506:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760757.068518:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25976832 i_blocks 51207 00000020:00200000:2.0:1789760757.068521:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760757.068528:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760757.068553:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25976832, count: 4096 00000080:00200000:2.0:1789760757.068560:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25976832, bytes: 4096 00000080:00200000:2.0:1789760757.068567:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760757.068583:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760757.068585:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25976832, 25980928) 0 0 00000080:00200000:2.0:1789760757.068589:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760757.068603:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6342, 6342] 00020000:00200000:2.0:1789760757.068611:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25976832 chunk: [25976832, 25980928) 25980928, 4096 00020000:00200000:2.0:1789760757.068614:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760757.068629:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6342, 6342] 00000008:00010000:2.0:1789760757.068645:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760757.068650:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760757.068660:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25976832, 25980928) append 0 append lockpos 0 00000080:00200000:2.0:1789760757.068671:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25976832=0x18c6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760757.068718:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760757.068721:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25980928 00000020:00010000:2.0:1789760757.068728:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760757.068731:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760757.068745:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760757.068749:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760757.068752:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760757.068759:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760757.068765:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760757.068779:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760757.068856:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 25976832<>25980928 00000100:00100000:3.0:1789760757.069029:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512673536:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760757.109419:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760757.109431:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760757.109440:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760757.109458:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760757.109487:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512673536:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760757.109528:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6342, 6342] 00000080:00200000:2.0:1789760757.109541:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760757.109546:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760757.109557:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25980928 i_blocks 51207 00000020:00200000:2.0:1789760757.109560:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760757.109567:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760757.109593:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25980928, count: 4096 00000080:00200000:2.0:1789760757.109613:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25980928, bytes: 4096 00000080:00200000:2.0:1789760757.109622:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760757.109630:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760757.109635:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25980928, 25985024) 0 0 00000080:00200000:2.0:1789760757.109639:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760757.109652:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6343, 6343] 00020000:00200000:2.0:1789760757.109658:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25980928 chunk: [25980928, 25985024) 25985024, 4096 00020000:00200000:2.0:1789760757.109662:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760757.109678:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6343, 6343] 00000008:00010000:2.0:1789760757.109692:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760757.109696:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760757.109705:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25980928, 25985024) append 0 append lockpos 0 00000080:00200000:2.0:1789760757.109715:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25980928=0x18c7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760757.109757:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760757.109759:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25985024 00000020:00010000:2.0:1789760757.109766:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760757.109769:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760757.109771:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760757.109774:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760757.109777:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760757.109783:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760757.110028:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760757.110043:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760757.110179:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 25980928<>25985024 00000100:00100000:2.0:1789760757.110217:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760757.110240:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512673664:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760757.154569:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973662 gen 3 00000008:00010000:3.0:1789760757.154616:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760757.154623:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760757.154630:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760757.154734:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760757.154759:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512673664:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760757.158316:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6343, 6343] 00000080:00200000:0.0:1789760757.158335:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760757.158340:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760757.158403:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25985024 i_blocks 51207 00000020:00200000:0.0:1789760757.158408:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760757.158420:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760757.158445:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25985024, count: 4096 00000080:00200000:0.0:1789760757.158453:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25985024, bytes: 4096 00000080:00200000:0.0:1789760757.158464:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760757.158471:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760757.158473:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25985024, 25989120) 0 0 00000080:00200000:0.0:1789760757.158477:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760757.158490:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6344, 6344] 00020000:00200000:0.0:1789760757.158498:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25985024 chunk: [25985024, 25989120) 25989120, 4096 00020000:00200000:0.0:1789760757.158501:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760757.158518:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6344, 6344] 00000008:00010000:0.0:1789760757.158532:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760757.158536:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760757.158545:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25985024, 25989120) append 0 append lockpos 0 00000080:00200000:0.0:1789760757.158556:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25985024=0x18c8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760757.158605:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760757.158608:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25989120 00000020:00010000:0.0:1789760757.158614:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760757.158617:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760757.158620:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760757.158623:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760757.158625:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760757.158631:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760757.158636:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760757.158648:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cca80 00000008:00100000:0.0:1789760757.158724:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148a80 - object 0x240000400:5 offset 25985024<>25989120 00000100:00100000:0.0:1789760757.158765:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512673792:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760757.224268:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973663 gen 3 00000008:00010000:0.0:1789760757.224304:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:0.0:1789760757.224321:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.224331:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.224358:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:0.0:1789760757.224381:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512673792:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760757.224472:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6344, 6344] 00000080:00200000:3.0:1789760757.224485:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760757.224491:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760757.224502:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25989120 i_blocks 51207 00000020:00200000:3.0:1789760757.224506:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760757.224513:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760757.224540:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25989120, count: 4096 00000080:00200000:3.0:1789760757.224546:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25989120, bytes: 4096 00000080:00200000:3.0:1789760757.224553:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760757.224560:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760757.224562:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25989120, 25993216) 0 0 00000080:00200000:3.0:1789760757.224565:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760757.224578:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6345, 6345] 00020000:00200000:3.0:1789760757.224584:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25989120 chunk: [25989120, 25993216) 25993216, 4096 00020000:00200000:3.0:1789760757.224588:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760757.224604:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6345, 6345] 00000008:00010000:3.0:1789760757.224617:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760757.224622:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760757.224632:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25989120, 25993216) append 0 append lockpos 0 00000080:00200000:3.0:1789760757.224642:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25989120=0x18c9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760757.224695:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760757.224698:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25993216 00000020:00010000:3.0:1789760757.224708:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760757.224711:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760757.224715:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760757.224718:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760757.224721:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760757.224727:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760757.224732:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760757.224747:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760757.225195:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad880 - object 0x240000400:5 offset 25989120<>25993216 00000100:00100000:2.0:1789760757.225247:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760757.225973:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512673920:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760757.267549:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973664 gen 3 00000008:00010000:3.0:1789760757.267663:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760757.267669:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760757.267679:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760757.268364:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760757.268389:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512673920:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760757.268508:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6345, 6345] 00000080:00200000:2.0:1789760757.268521:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760757.268526:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760757.268538:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25993216 i_blocks 51207 00000020:00200000:2.0:1789760757.268541:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760757.268550:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760757.268575:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25993216, count: 4096 00000080:00200000:2.0:1789760757.268650:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25993216, bytes: 4096 00000080:00200000:2.0:1789760757.268660:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760757.268668:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760757.268670:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25993216, 25997312) 0 0 00000080:00200000:2.0:1789760757.268675:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760757.268850:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6346, 6346] 00020000:00200000:2.0:1789760757.268858:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25993216 chunk: [25993216, 25997312) 25997312, 4096 00020000:00200000:2.0:1789760757.268862:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760757.268877:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6346, 6346] 00000008:00010000:2.0:1789760757.268890:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760757.268895:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760757.268921:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25993216, 25997312) append 0 append lockpos 0 00000080:00200000:2.0:1789760757.268933:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25993216=0x18ca000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760757.268975:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760757.269040:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 25997312 00000020:00010000:2.0:1789760757.269049:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760757.269052:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760757.269055:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760757.269059:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760757.269061:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760757.269069:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760757.269075:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760757.269090:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760757.269168:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 25993216<>25997312 00000100:00100000:3.0:1789760757.269227:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512674048:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760757.307763:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973665 gen 3 00000008:00010000:3.0:1789760757.307799:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760757.307932:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760757.307942:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760757.308000:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760757.308025:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512674048:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760757.308140:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6346, 6346] 00000080:00200000:2.0:1789760757.308152:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760757.308157:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760757.308167:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 25997312 i_blocks 51207 00000020:00200000:2.0:1789760757.308171:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760757.308179:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760757.308263:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 25997312, count: 4096 00000080:00200000:2.0:1789760757.308338:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 25997312, bytes: 4096 00000080:00200000:2.0:1789760757.308348:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760757.308465:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760757.308467:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [25997312, 26001408) 0 0 00000080:00200000:2.0:1789760757.308471:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760757.308485:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6347, 6347] 00020000:00200000:2.0:1789760757.308492:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 25997312 chunk: [25997312, 26001408) 26001408, 4096 00020000:00200000:2.0:1789760757.308496:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760757.308510:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6347, 6347] 00000008:00010000:2.0:1789760757.308534:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760757.308539:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760757.308547:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [25997312, 26001408) append 0 append lockpos 0 00000080:00200000:2.0:1789760757.308558:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=25997312=0x18cb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760757.308599:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760757.308601:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26001408 00000020:00010000:2.0:1789760757.308607:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760757.308610:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760757.308613:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760757.308616:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760757.308619:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760757.308625:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760757.308631:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760757.308645:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000008:00100000:2.0:1789760757.308726:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 25997312<>26001408 00000100:00100000:3.0:1789760757.308788:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512674176:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760757.350452:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973666 gen 3 00000008:00010000:3.0:1789760757.350488:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760757.350495:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760757.350504:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760757.350568:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760757.350593:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512674176:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760757.350671:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6347, 6347] 00000080:00200000:0.0:1789760757.350686:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760757.350692:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760757.350770:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26001408 i_blocks 51207 00000020:00200000:0.0:1789760757.350774:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760757.350782:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760757.350955:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26001408, count: 4096 00000080:00200000:0.0:1789760757.350963:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26001408, bytes: 4096 00000080:00200000:0.0:1789760757.350972:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760757.351045:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760757.351048:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26001408, 26005504) 0 0 00000080:00200000:0.0:1789760757.351052:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760757.351065:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6348, 6348] 00020000:00200000:0.0:1789760757.351072:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26001408 chunk: [26001408, 26005504) 26005504, 4096 00020000:00200000:0.0:1789760757.351076:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760757.351090:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6348, 6348] 00000008:00010000:0.0:1789760757.351103:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760757.351108:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760757.351116:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26001408, 26005504) append 0 append lockpos 0 00000080:00200000:0.0:1789760757.351127:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26001408=0x18cc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760757.351164:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760757.351166:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26005504 00000020:00010000:0.0:1789760757.351172:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760757.351175:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760757.351178:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760757.351181:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760757.351184:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760757.351190:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760757.351196:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760757.351209:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cca80 00000008:00100000:0.0:1789760757.351371:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148a80 - object 0x240000400:5 offset 26001408<>26005504 00000100:00100000:1.0:1789760757.351590:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512674304:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760757.392338:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973667 gen 3 00000008:00010000:1.0:1789760757.392375:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760757.392381:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760757.392390:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760757.392451:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760757.392476:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512674304:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760757.393519:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6348, 6348] 00000080:00200000:3.0:1789760757.393532:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760757.393537:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760757.393549:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26005504 i_blocks 51207 00000020:00200000:3.0:1789760757.393553:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760757.393563:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760757.393586:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26005504, count: 4096 00000080:00200000:3.0:1789760757.393591:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26005504, bytes: 4096 00000080:00200000:3.0:1789760757.393600:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760757.393607:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760757.393609:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26005504, 26009600) 0 0 00000080:00200000:3.0:1789760757.393613:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760757.393625:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6349, 6349] 00020000:00200000:3.0:1789760757.393631:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26005504 chunk: [26005504, 26009600) 26009600, 4096 00020000:00200000:3.0:1789760757.393634:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760757.393648:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6349, 6349] 00000008:00010000:3.0:1789760757.393660:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760757.393665:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760757.393673:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26005504, 26009600) append 0 append lockpos 0 00000080:00200000:3.0:1789760757.393682:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26005504=0x18cd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760757.393796:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760757.393799:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26009600 00000020:00010000:3.0:1789760757.393805:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760757.393808:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760757.393811:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760757.393814:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760757.393817:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760757.393823:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760757.393828:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760757.393844:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760757.394165:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad880 - object 0x240000400:5 offset 26005504<>26009600 00000100:00100000:2.0:1789760757.394223:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512674432:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760757.439334:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973668 gen 3 00000008:00010000:3.0:1789760757.439371:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760757.439377:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760757.439386:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760757.439403:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760757.439424:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512674432:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760757.439526:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6349, 6349] 00000080:00200000:2.0:1789760757.439541:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760757.439546:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760757.439559:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26009600 i_blocks 51207 00000020:00200000:2.0:1789760757.439562:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760757.439570:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760757.439598:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26009600, count: 4096 00000080:00200000:2.0:1789760757.439605:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26009600, bytes: 4096 00000080:00200000:2.0:1789760757.439614:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760757.439622:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760757.439624:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26009600, 26013696) 0 0 00000080:00200000:2.0:1789760757.439629:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760757.439643:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6350, 6350] 00020000:00200000:2.0:1789760757.439652:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26009600 chunk: [26009600, 26013696) 26013696, 4096 00020000:00200000:2.0:1789760757.439655:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760757.439671:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6350, 6350] 00000008:00010000:2.0:1789760757.439685:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760757.439689:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760757.439710:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26009600, 26013696) append 0 append lockpos 0 00000080:00200000:2.0:1789760757.439720:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26009600=0x18ce000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760757.439761:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760757.439763:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26013696 00000020:00010000:2.0:1789760757.439769:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760757.439772:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760757.439775:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760757.439779:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760757.439782:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760757.439788:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760757.439793:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760757.439805:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760757.440137:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 26009600<>26013696 00000100:00100000:2.0:1789760757.440173:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512674560:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760757.478646:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973669 gen 3 00000008:00010000:3.0:1789760757.478682:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760757.478689:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760757.478705:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760757.478778:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760757.478803:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512674560:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760757.481426:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6350, 6350] 00000080:00200000:0.0:1789760757.481443:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760757.481448:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760757.481461:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26013696 i_blocks 51207 00000020:00200000:0.0:1789760757.481465:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760757.481476:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760757.481500:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26013696, count: 4096 00000080:00200000:0.0:1789760757.481506:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26013696, bytes: 4096 00000080:00200000:0.0:1789760757.481516:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760757.481524:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760757.481525:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26013696, 26017792) 0 0 00000080:00200000:0.0:1789760757.481529:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760757.481540:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6351, 6351] 00020000:00200000:0.0:1789760757.481546:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26013696 chunk: [26013696, 26017792) 26017792, 4096 00020000:00200000:0.0:1789760757.481550:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760757.481563:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6351, 6351] 00000008:00010000:0.0:1789760757.481575:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760757.481579:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760757.481589:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26013696, 26017792) append 0 append lockpos 0 00000080:00200000:0.0:1789760757.481599:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26013696=0x18cf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760757.481643:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760757.481645:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26017792 00000020:00010000:0.0:1789760757.481651:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760757.481654:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760757.481656:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760757.481659:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760757.481663:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760757.481668:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760757.481672:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760757.481693:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc540 00000008:00100000:0.0:1789760757.482673:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148e00 - object 0x240000400:5 offset 26013696<>26017792 00000100:00100000:1.0:1789760757.482737:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512674688:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760757.526514:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973670 gen 3 00000008:00010000:1.0:1789760757.526548:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760757.526554:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760757.526563:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760757.526589:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760757.526612:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512674688:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760757.526718:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6351, 6351] 00000080:00200000:3.0:1789760757.526735:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760757.526740:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760757.526750:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26017792 i_blocks 51207 00000020:00200000:3.0:1789760757.526753:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760757.526761:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760757.526785:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26017792, count: 4096 00000080:00200000:3.0:1789760757.526791:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26017792, bytes: 4096 00000080:00200000:3.0:1789760757.526801:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760757.526807:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760757.526810:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26017792, 26021888) 0 0 00000080:00200000:3.0:1789760757.526813:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760757.526825:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6352, 6352] 00020000:00200000:3.0:1789760757.526831:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26017792 chunk: [26017792, 26021888) 26021888, 4096 00020000:00200000:3.0:1789760757.526834:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760757.526847:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6352, 6352] 00000008:00010000:3.0:1789760757.526861:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760757.526866:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760757.526875:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26017792, 26021888) append 0 append lockpos 0 00000080:00200000:3.0:1789760757.526885:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26017792=0x18d0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760757.526964:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760757.526968:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26021888 00000020:00010000:3.0:1789760757.526974:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760757.526977:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760757.526980:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760757.526982:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760757.526985:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760757.526990:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760757.526996:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760757.527009:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760757.527084:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad880 - object 0x240000400:5 offset 26017792<>26021888 00000100:00100000:3.0:1789760757.527121:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760757.527133:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512674816:192.168.203.156@tcp:4:dd.0 00010000:00010000:3.0:1789760757.533797:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:3.0:1789760757.562618:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760757.562628:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760757.562636:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760757.562753:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760757.562778:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512674816:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760757.566043:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6352, 6352] 00000080:00200000:1.0:1789760757.566063:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.566068:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.566081:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26021888 i_blocks 51207 00000020:00200000:1.0:1789760757.566086:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.566098:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.566122:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26021888, count: 4096 00000080:00200000:1.0:1789760757.566129:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26021888, bytes: 4096 00000080:00200000:1.0:1789760757.566140:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.566149:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.566151:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26021888, 26025984) 0 0 00000080:00200000:1.0:1789760757.566155:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.566167:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6353, 6353] 00020000:00200000:1.0:1789760757.566173:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26021888 chunk: [26021888, 26025984) 26025984, 4096 00020000:00200000:1.0:1789760757.566178:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.566194:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6353, 6353] 00000008:00010000:1.0:1789760757.566209:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.566214:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760757.566223:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26021888, 26025984) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.566234:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26021888=0x18d1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.566278:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.566281:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26025984 00000020:00010000:1.0:1789760757.566286:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.566289:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.566291:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.566294:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.566297:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.566313:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.566319:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.566333:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760757.566415:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 26021888<>26025984 00000100:00100000:0.0:1789760757.566476:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512674944:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760757.597153:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973672 gen 3 00000008:00010000:0.0:1789760757.597198:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760757.597204:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.597212:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.597227:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000080:00200000:1.0:1789760757.597244:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6353, 6353] 00000100:00100000:0.0:1789760757.597250:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512674944:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760757.597257:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.597261:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.597272:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26025984 i_blocks 51207 00000020:00200000:1.0:1789760757.597275:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.597283:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.597315:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26025984, count: 4096 00000080:00200000:1.0:1789760757.597320:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26025984, bytes: 4096 00000080:00200000:1.0:1789760757.597327:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.597332:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.597333:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26025984, 26030080) 0 0 00000080:00200000:1.0:1789760757.597336:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.597346:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6354, 6354] 00020000:00200000:1.0:1789760757.597352:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26025984 chunk: [26025984, 26030080) 26030080, 4096 00020000:00200000:1.0:1789760757.597354:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.597367:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6354, 6354] 00000008:00010000:1.0:1789760757.597376:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.597380:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760757.597388:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26025984, 26030080) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.597396:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26025984=0x18d2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.597528:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.597531:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26030080 00000020:00010000:1.0:1789760757.597538:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.597541:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.597544:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.597547:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.597550:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.597556:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.597561:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.597573:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760757.597647:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 26025984<>26030080 00000100:00100000:0.0:1789760757.597836:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512675072:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760757.623010:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973673 gen 3 00000008:00010000:0.0:1789760757.623044:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760757.623050:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.623058:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.623081:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760757.623103:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512675072:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760757.623374:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6354, 6354] 00000080:00200000:1.0:1789760757.623385:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.623388:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.623398:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26030080 i_blocks 51207 00000020:00200000:1.0:1789760757.623400:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.623406:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.623427:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26030080, count: 4096 00000080:00200000:1.0:1789760757.623431:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26030080, bytes: 4096 00000080:00200000:1.0:1789760757.623437:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.623442:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.623443:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26030080, 26034176) 0 0 00000080:00200000:1.0:1789760757.623446:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.623455:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6355, 6355] 00020000:00200000:1.0:1789760757.623460:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26030080 chunk: [26030080, 26034176) 26034176, 4096 00020000:00200000:1.0:1789760757.623462:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.623473:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6355, 6355] 00000008:00010000:1.0:1789760757.623483:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.623486:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760757.623493:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26030080, 26034176) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.623501:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26030080=0x18d3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.623533:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.623535:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26034176 00000020:00010000:1.0:1789760757.623540:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.623542:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.623544:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.623546:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.623548:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.623552:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.623556:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.623565:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760757.623626:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97100 - object 0x240000400:5 offset 26030080<>26034176 00000100:00100000:0.0:1789760757.623675:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512675200:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760757.650748:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973674 gen 3 00000008:00010000:0.0:1789760757.650781:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760757.650842:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.650851:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.650871:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760757.650893:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512675200:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760757.651518:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6355, 6355] 00000080:00200000:1.0:1789760757.651532:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.651537:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.651549:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26034176 i_blocks 51207 00000020:00200000:1.0:1789760757.651552:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.651560:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.651585:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26034176, count: 4096 00000080:00200000:1.0:1789760757.651592:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26034176, bytes: 4096 00000080:00200000:1.0:1789760757.651600:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.651607:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.651609:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26034176, 26038272) 0 0 00000080:00200000:1.0:1789760757.651613:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.651626:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6356, 6356] 00020000:00200000:1.0:1789760757.651632:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26034176 chunk: [26034176, 26038272) 26038272, 4096 00020000:00200000:1.0:1789760757.651635:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.651649:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6356, 6356] 00000008:00010000:1.0:1789760757.651661:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.651666:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760757.651674:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26034176, 26038272) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.651684:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26034176=0x18d4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.651729:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.651731:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26038272 00000020:00010000:1.0:1789760757.651737:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.651741:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.651743:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.651746:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.651749:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.651755:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.651760:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.651785:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760757.652062:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 26034176<>26038272 00000100:00100000:0.0:1789760757.652161:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512675328:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760757.691345:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973675 gen 3 00000008:00010000:0.0:1789760757.691391:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760757.691398:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.691415:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.691432:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760757.691454:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512675328:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760757.691602:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6356, 6356] 00000080:00200000:1.0:1789760757.691616:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.691621:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.691633:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26038272 i_blocks 51207 00000020:00200000:1.0:1789760757.691636:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.691643:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.691668:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26038272, count: 4096 00000080:00200000:1.0:1789760757.691675:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26038272, bytes: 4096 00000080:00200000:1.0:1789760757.691683:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.691690:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.691692:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26038272, 26042368) 0 0 00000080:00200000:1.0:1789760757.691696:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.691709:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6357, 6357] 00020000:00200000:1.0:1789760757.691716:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26038272 chunk: [26038272, 26042368) 26042368, 4096 00020000:00200000:1.0:1789760757.691719:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.691734:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6357, 6357] 00000008:00010000:1.0:1789760757.691748:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.691753:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760757.691764:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26038272, 26042368) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.691776:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26038272=0x18d5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.691831:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.691834:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26042368 00000020:00010000:1.0:1789760757.691841:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.691844:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.691847:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.691850:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.692082:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.692090:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.692097:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.692115:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760757.692198:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 26038272<>26042368 00000100:00100000:0.0:1789760757.692277:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760757.692291:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512675456:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760757.725863:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973676 gen 3 00000008:00010000:0.0:1789760757.725896:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760757.725925:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.725933:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.725946:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760757.725968:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512675456:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760757.726461:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6357, 6357] 00000080:00200000:1.0:1789760757.726476:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.726481:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.726492:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26042368 i_blocks 51207 00000020:00200000:1.0:1789760757.726496:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.726503:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.726528:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26042368, count: 4096 00000080:00200000:1.0:1789760757.726534:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26042368, bytes: 4096 00000080:00200000:1.0:1789760757.726543:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.726549:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.726551:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26042368, 26046464) 0 0 00000080:00200000:1.0:1789760757.726555:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.726568:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6358, 6358] 00020000:00200000:1.0:1789760757.726574:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26042368 chunk: [26042368, 26046464) 26046464, 4096 00020000:00200000:1.0:1789760757.726577:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.726593:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6358, 6358] 00000008:00010000:1.0:1789760757.726606:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.726611:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760757.726620:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26042368, 26046464) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.726630:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26042368=0x18d6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.726672:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.726675:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26046464 00000020:00010000:1.0:1789760757.726681:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.726685:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.726688:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.726691:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.726705:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.726712:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.726717:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.726732:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760757.726818:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97100 - object 0x240000400:5 offset 26042368<>26046464 00000100:00100000:0.0:1789760757.727158:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512675584:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760757.779409:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973677 gen 3 00000008:00010000:0.0:1789760757.779449:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760757.779455:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.779463:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.779520:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760757.779541:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512675584:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760757.779611:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6358, 6358] 00000080:00200000:1.0:1789760757.779624:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.779629:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.779642:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26046464 i_blocks 51207 00000020:00200000:1.0:1789760757.779645:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.779654:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.779684:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26046464, count: 4096 00000080:00200000:1.0:1789760757.779690:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26046464, bytes: 4096 00000080:00200000:1.0:1789760757.779745:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.779754:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.779757:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26046464, 26050560) 0 0 00000080:00200000:1.0:1789760757.779761:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.779775:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6359, 6359] 00020000:00200000:1.0:1789760757.780015:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26046464 chunk: [26046464, 26050560) 26050560, 4096 00020000:00200000:1.0:1789760757.780021:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.780041:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6359, 6359] 00000008:00010000:1.0:1789760757.780054:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.780059:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760757.780068:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26046464, 26050560) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.780078:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26046464=0x18d7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.780122:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.780125:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26050560 00000020:00010000:1.0:1789760757.780132:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.780135:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.780138:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.780142:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.780145:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.780151:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.780157:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.780172:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760757.780255:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 26046464<>26050560 00000100:00100000:0.0:1789760757.780327:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512675712:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760757.817262:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973678 gen 3 00000008:00010000:0.0:1789760757.817296:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760757.817311:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.817319:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.817337:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760757.817357:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512675712:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760757.817454:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6359, 6359] 00000080:00200000:1.0:1789760757.817466:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.817471:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.817482:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26050560 i_blocks 51207 00000020:00200000:1.0:1789760757.817485:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.817493:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.817516:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26050560, count: 4096 00000080:00200000:1.0:1789760757.817521:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26050560, bytes: 4096 00000080:00200000:1.0:1789760757.817528:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.817535:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.817537:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26050560, 26054656) 0 0 00000080:00200000:1.0:1789760757.817540:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.817552:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6360, 6360] 00020000:00200000:1.0:1789760757.817559:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26050560 chunk: [26050560, 26054656) 26054656, 4096 00020000:00200000:1.0:1789760757.817563:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.817577:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6360, 6360] 00000008:00010000:1.0:1789760757.817589:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.817593:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760757.817601:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26050560, 26054656) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.817610:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26050560=0x18d8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.817647:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.817650:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26054656 00000020:00010000:1.0:1789760757.817656:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.817659:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.817662:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.817665:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.817668:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.817673:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.817678:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.817689:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760757.817765:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 26050560<>26054656 00000100:00100000:0.0:1789760757.817814:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512675840:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760757.856487:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760757.856498:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.856507:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.856576:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760757.856599:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512675840:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760757.857055:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6360, 6360] 00000080:00200000:1.0:1789760757.857066:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.857071:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.857081:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26054656 i_blocks 51207 00000020:00200000:1.0:1789760757.857084:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.857093:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.857118:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26054656, count: 4096 00000080:00200000:1.0:1789760757.857125:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26054656, bytes: 4096 00000080:00200000:1.0:1789760757.857132:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.857139:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.857141:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26054656, 26058752) 0 0 00000080:00200000:1.0:1789760757.857145:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.857156:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6361, 6361] 00020000:00200000:1.0:1789760757.857164:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26054656 chunk: [26054656, 26058752) 26058752, 4096 00020000:00200000:1.0:1789760757.857168:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.857182:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6361, 6361] 00000008:00010000:1.0:1789760757.857195:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.857200:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760757.857209:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26054656, 26058752) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.857219:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26054656=0x18d9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.857263:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.857265:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26058752 00000020:00010000:1.0:1789760757.857271:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.857274:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.857276:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.857279:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.857282:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.857288:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.857293:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.857318:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760757.857395:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95c00 - object 0x240000400:5 offset 26054656<>26058752 00000100:00100000:0.0:1789760757.857459:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512675968:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760757.894025:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973680 gen 3 00000008:00010000:0.0:1789760757.894075:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760757.894081:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.894088:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.894111:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760757.894133:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512675968:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760757.894271:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6361, 6361] 00000080:00200000:1.0:1789760757.894283:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.894288:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.894299:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26058752 i_blocks 51207 00000020:00200000:1.0:1789760757.894313:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.894321:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.894348:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26058752, count: 4096 00000080:00200000:1.0:1789760757.894355:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26058752, bytes: 4096 00000080:00200000:1.0:1789760757.894362:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.894368:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.894370:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26058752, 26062848) 0 0 00000080:00200000:1.0:1789760757.894374:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.894388:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6362, 6362] 00020000:00200000:1.0:1789760757.894394:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26058752 chunk: [26058752, 26062848) 26062848, 4096 00020000:00200000:1.0:1789760757.894397:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.894415:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6362, 6362] 00000008:00010000:1.0:1789760757.894428:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.894433:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760757.894442:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26058752, 26062848) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.894453:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26058752=0x18da000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.894496:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.894499:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26062848 00000020:00010000:1.0:1789760757.894506:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.894509:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.894512:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.894515:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.894518:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.894524:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.894530:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.894544:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760757.894621:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 26058752<>26062848 00000100:00100000:0.0:1789760757.895080:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512676096:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760757.930078:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973681 gen 3 00000008:00010000:0.0:1789760757.930112:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760757.930119:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.930127:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.930145:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760757.930165:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512676096:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760757.930254:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6362, 6362] 00000080:00200000:1.0:1789760757.930265:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.930270:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.930282:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26062848 i_blocks 51207 00000020:00200000:1.0:1789760757.930285:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.930293:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.930326:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26062848, count: 4096 00000080:00200000:1.0:1789760757.930332:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26062848, bytes: 4096 00000080:00200000:1.0:1789760757.930340:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.930346:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.930348:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26062848, 26066944) 0 0 00000080:00200000:1.0:1789760757.930351:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.930363:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6363, 6363] 00020000:00200000:1.0:1789760757.930370:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26062848 chunk: [26062848, 26066944) 26066944, 4096 00020000:00200000:1.0:1789760757.930374:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.930389:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6363, 6363] 00000008:00010000:1.0:1789760757.930401:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.930406:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760757.930414:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26062848, 26066944) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.930425:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26062848=0x18db000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.930465:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.930468:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26066944 00000020:00010000:1.0:1789760757.930475:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.930478:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.930481:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.930484:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.930486:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.930491:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.930496:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.930508:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760757.930581:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 26062848<>26066944 00000100:00100000:0.0:1789760757.930661:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760757.930674:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512676224:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760757.961745:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973682 gen 3 00000008:00010000:0.0:1789760757.961800:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760757.961807:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.962081:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.962102:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760757.962126:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512676224:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760757.962220:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6363, 6363] 00000080:00200000:1.0:1789760757.962233:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.962237:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.962250:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26066944 i_blocks 51207 00000020:00200000:1.0:1789760757.962253:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.962261:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.962286:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26066944, count: 4096 00000080:00200000:1.0:1789760757.962292:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26066944, bytes: 4096 00000080:00200000:1.0:1789760757.962299:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.962316:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.962318:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26066944, 26071040) 0 0 00000080:00200000:1.0:1789760757.962322:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.962339:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6364, 6364] 00020000:00200000:1.0:1789760757.962347:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26066944 chunk: [26066944, 26071040) 26071040, 4096 00020000:00200000:1.0:1789760757.962350:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.962365:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6364, 6364] 00000008:00010000:1.0:1789760757.962378:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.962382:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760757.962392:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26066944, 26071040) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.962402:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26066944=0x18dc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.962445:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.962448:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26071040 00000020:00010000:1.0:1789760757.962454:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.962458:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.962461:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.962464:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.962466:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.962473:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.962478:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.962492:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760757.962569:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94700 - object 0x240000400:5 offset 26066944<>26071040 00000100:00100000:0.0:1789760757.962782:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512676352:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760757.989844:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973683 gen 3 00000008:00010000:0.0:1789760757.989880:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760757.989887:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760757.989895:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760757.989937:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760757.989959:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512676352:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760757.990105:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6364, 6364] 00000080:00200000:1.0:1789760757.990120:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760757.990125:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760757.990136:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26071040 i_blocks 51207 00000020:00200000:1.0:1789760757.990140:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760757.990147:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760757.990175:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26071040, count: 4096 00000080:00200000:1.0:1789760757.990181:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26071040, bytes: 4096 00000080:00200000:1.0:1789760757.990189:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760757.990196:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760757.990198:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26071040, 26075136) 0 0 00000080:00200000:1.0:1789760757.990202:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760757.990215:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6365, 6365] 00020000:00200000:1.0:1789760757.990222:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26071040 chunk: [26071040, 26075136) 26075136, 4096 00020000:00200000:1.0:1789760757.990224:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760757.990237:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6365, 6365] 00000008:00010000:1.0:1789760757.990248:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760757.990251:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760757.990258:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26071040, 26075136) append 0 append lockpos 0 00000080:00200000:1.0:1789760757.990266:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26071040=0x18dd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760757.990314:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760757.990316:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26075136 00000020:00010000:1.0:1789760757.990323:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760757.990326:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.990329:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760757.990332:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760757.990335:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760757.990341:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760757.990346:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760757.990413:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760757.990490:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 26071040<>26075136 00000100:00100000:1.0:1789760757.990542:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512676480:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760758.019472:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973684 gen 3 00000008:00010000:1.0:1789760758.019508:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760758.019515:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760758.019523:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760758.019546:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000080:00200000:3.0:1789760758.019700:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6365, 6365] 00000080:00200000:3.0:1789760758.019713:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.019722:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.019734:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26075136 i_blocks 51207 00000020:00200000:3.0:1789760758.019738:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.019747:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.019776:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26075136, count: 4096 00000080:00200000:3.0:1789760758.019787:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26075136, bytes: 4096 00000080:00200000:3.0:1789760758.019795:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.019802:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.019804:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26075136, 26079232) 0 0 00000080:00200000:3.0:1789760758.019808:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.019821:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6366, 6366] 00020000:00200000:3.0:1789760758.019828:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26075136 chunk: [26075136, 26079232) 26079232, 4096 00020000:00200000:3.0:1789760758.019831:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.019846:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6366, 6366] 00000008:00010000:3.0:1789760758.019859:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.019864:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760758.019872:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26075136, 26079232) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.019884:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26075136=0x18de000, pages 1 (max 131072), lockless, parallel 00000100:00100000:1.0:1789760758.021681:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512676480:192.168.203.156@tcp:4:dd.0 00000020:00200000:3.0:1789760758.022319:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.022324:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26079232 00000020:00010000:3.0:1789760758.022335:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.022339:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000008:00010000:1.0:1789760758.022341:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00010000:3.0:1789760758.022344:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.022349:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.022352:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.022369:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.022378:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00100000:1.0:1789760758.022427:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 26075136<>26079232 00000100:00100000:0.0:1789760758.022551:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512676608:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760758.068636:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:0.0:1789760758.068648:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760758.068658:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760758.068679:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:0.0:1789760758.068704:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512676608:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760758.068721:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6366, 6366] 00000080:00200000:3.0:1789760758.068734:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.068740:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.068752:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26079232 i_blocks 51207 00000020:00200000:3.0:1789760758.068757:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.068765:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.068791:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26079232, count: 4096 00000080:00200000:3.0:1789760758.068797:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26079232, bytes: 4096 00000080:00200000:3.0:1789760758.068806:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.068813:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.068815:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26079232, 26083328) 0 0 00000080:00200000:3.0:1789760758.068819:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.068833:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6367, 6367] 00020000:00200000:3.0:1789760758.068839:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26079232 chunk: [26079232, 26083328) 26083328, 4096 00020000:00200000:3.0:1789760758.068842:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.068856:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6367, 6367] 00000008:00010000:3.0:1789760758.068870:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.068874:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760758.068883:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26079232, 26083328) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.068894:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26079232=0x18df000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.068984:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.068987:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26083328 00000020:00010000:3.0:1789760758.068993:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.068997:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.069000:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.069003:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.069006:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.069012:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.069019:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.069030:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760758.069106:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 26079232<>26083328 00000100:00100000:2.0:1789760758.069219:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512676736:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760758.122210:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973686 gen 3 00000008:00010000:3.0:1789760758.122256:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:3.0:1789760758.122262:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760758.122271:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760758.122405:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:3.0:1789760758.122436:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512676736:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760758.122682:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6367, 6367] 00000080:00200000:1.0:1789760758.122695:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760758.122700:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760758.122713:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26083328 i_blocks 51207 00000020:00200000:1.0:1789760758.122716:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760758.122726:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760758.122756:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26083328, count: 4096 00000080:00200000:1.0:1789760758.122762:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26083328, bytes: 4096 00000080:00200000:1.0:1789760758.122771:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760758.122778:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760758.122781:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26083328, 26087424) 0 0 00000080:00200000:1.0:1789760758.122785:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760758.122852:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6368, 6368] 00020000:00200000:1.0:1789760758.122860:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26083328 chunk: [26083328, 26087424) 26087424, 4096 00020000:00200000:1.0:1789760758.122863:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760758.122880:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6368, 6368] 00000008:00010000:1.0:1789760758.122893:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760758.122929:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760758.122940:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26083328, 26087424) append 0 append lockpos 0 00000080:00200000:1.0:1789760758.122951:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26083328=0x18e0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760758.124334:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760758.124337:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26087424 00000020:00010000:1.0:1789760758.124347:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760758.124351:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.124354:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.124357:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760758.124360:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760758.124368:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760758.124374:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760758.124397:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760758.124481:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94380 - object 0x240000400:5 offset 26083328<>26087424 00000100:00100000:0.0:1789760758.124839:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512676864:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760758.176843:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973687 gen 3 00000008:00010000:0.0:1789760758.176895:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760758.176927:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760758.176937:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760758.176960:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760758.176983:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512676864:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760758.179166:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6368, 6368] 00000080:00200000:1.0:1789760758.179186:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760758.179192:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760758.179216:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26087424 i_blocks 51207 00000020:00200000:1.0:1789760758.179220:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760758.179231:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760758.179256:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26087424, count: 4096 00000080:00200000:1.0:1789760758.179263:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26087424, bytes: 4096 00000080:00200000:1.0:1789760758.179273:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760758.179281:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760758.179283:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26087424, 26091520) 0 0 00000080:00200000:1.0:1789760758.179286:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760758.179307:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6369, 6369] 00020000:00200000:1.0:1789760758.179315:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26087424 chunk: [26087424, 26091520) 26091520, 4096 00020000:00200000:1.0:1789760758.179319:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760758.179346:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6369, 6369] 00000008:00010000:1.0:1789760758.179360:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760758.179365:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760758.179374:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26087424, 26091520) append 0 append lockpos 0 00000080:00200000:1.0:1789760758.179386:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26087424=0x18e1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760758.179434:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760758.179438:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26091520 00000020:00010000:1.0:1789760758.179444:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760758.179447:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.179450:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.179454:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760758.179456:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760758.179462:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760758.179468:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760758.179484:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760758.179563:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 26087424<>26091520 00000100:00100000:0.0:1789760758.183028:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760758.183060:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512676992:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760758.235841:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973688 gen 3 00000008:00010000:0.0:1789760758.235880:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760758.235887:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760758.235895:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760758.235955:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760758.235980:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512676992:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760758.237830:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6369, 6369] 00000080:00200000:1.0:1789760758.237848:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760758.237853:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760758.237865:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26091520 i_blocks 51207 00000020:00200000:1.0:1789760758.237869:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760758.237880:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760758.237931:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26091520, count: 4096 00000080:00200000:1.0:1789760758.237939:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26091520, bytes: 4096 00000080:00200000:1.0:1789760758.237949:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760758.237957:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760758.237959:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26091520, 26095616) 0 0 00000080:00200000:1.0:1789760758.237962:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760758.237975:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6370, 6370] 00020000:00200000:1.0:1789760758.237981:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26091520 chunk: [26091520, 26095616) 26095616, 4096 00020000:00200000:1.0:1789760758.237985:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760758.238000:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6370, 6370] 00000008:00010000:1.0:1789760758.238014:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760758.238018:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760758.238029:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26091520, 26095616) append 0 append lockpos 0 00000080:00200000:1.0:1789760758.238040:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26091520=0x18e2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760758.238461:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760758.238464:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26095616 00000020:00010000:1.0:1789760758.238471:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760758.238474:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.238477:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.238481:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760758.238484:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760758.238491:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760758.238497:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760758.238515:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760758.238594:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 26091520<>26095616 00000100:00100000:0.0:1789760758.238659:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760758.238675:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512677120:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760758.290150:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973689 gen 3 00000008:00010000:0.0:1789760758.290188:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760758.290196:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760758.290204:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760758.290224:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760758.290248:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512677120:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760758.290869:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6370, 6370] 00000080:00200000:1.0:1789760758.290882:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760758.290886:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760758.290898:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26095616 i_blocks 51207 00000020:00200000:1.0:1789760758.290926:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760758.290934:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760758.290957:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26095616, count: 4096 00000080:00200000:1.0:1789760758.290962:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26095616, bytes: 4096 00000080:00200000:1.0:1789760758.290972:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760758.290978:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760758.290980:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26095616, 26099712) 0 0 00000080:00200000:1.0:1789760758.290983:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760758.290995:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6371, 6371] 00020000:00200000:1.0:1789760758.291002:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26095616 chunk: [26095616, 26099712) 26099712, 4096 00020000:00200000:1.0:1789760758.291005:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760758.291018:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6371, 6371] 00000008:00010000:1.0:1789760758.291031:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760758.291034:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760758.291042:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26095616, 26099712) append 0 append lockpos 0 00000080:00200000:1.0:1789760758.291053:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26095616=0x18e3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760758.291112:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760758.291115:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26099712 00000020:00010000:1.0:1789760758.291121:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760758.291124:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.291127:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.291129:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760758.291132:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760758.291139:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760758.291144:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760758.291158:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760758.291235:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 26095616<>26099712 00000100:00100000:0.0:1789760758.291611:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760758.291629:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512677248:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760758.384685:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973690 gen 3 00000008:00010000:0.0:1789760758.384722:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760758.384728:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760758.384739:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760758.384758:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760758.384778:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512677248:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760758.385084:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6371, 6371] 00000080:00200000:1.0:1789760758.385097:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760758.385102:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760758.385113:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26099712 i_blocks 51207 00000020:00200000:1.0:1789760758.385116:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760758.385123:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760758.385145:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26099712, count: 4096 00000080:00200000:1.0:1789760758.385150:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26099712, bytes: 4096 00000080:00200000:1.0:1789760758.385157:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760758.385164:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760758.385166:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26099712, 26103808) 0 0 00000080:00200000:1.0:1789760758.385169:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760758.385180:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6372, 6372] 00020000:00200000:1.0:1789760758.385187:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26099712 chunk: [26099712, 26103808) 26103808, 4096 00020000:00200000:1.0:1789760758.385190:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760758.385203:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6372, 6372] 00000008:00010000:1.0:1789760758.385214:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760758.385218:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760758.385226:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26099712, 26103808) append 0 append lockpos 0 00000080:00200000:1.0:1789760758.385235:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26099712=0x18e4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760758.385271:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760758.385273:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26103808 00000020:00010000:1.0:1789760758.385279:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760758.385282:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.385284:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.385287:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760758.385290:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760758.385306:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760758.385311:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760758.385323:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760758.385387:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96680 - object 0x240000400:5 offset 26099712<>26103808 00000100:00100000:0.0:1789760758.385435:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512677376:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760758.426774:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760758.426785:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760758.426793:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760758.426822:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760758.426845:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512677376:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760758.427073:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6372, 6372] 00000080:00200000:1.0:1789760758.427086:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760758.427092:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760758.427104:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26103808 i_blocks 51207 00000020:00200000:1.0:1789760758.427107:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760758.427114:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760758.427137:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26103808, count: 4096 00000080:00200000:1.0:1789760758.427144:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26103808, bytes: 4096 00000080:00200000:1.0:1789760758.427151:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760758.427158:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760758.427160:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26103808, 26107904) 0 0 00000080:00200000:1.0:1789760758.427164:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760758.427176:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6373, 6373] 00020000:00200000:1.0:1789760758.427182:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26103808 chunk: [26103808, 26107904) 26107904, 4096 00020000:00200000:1.0:1789760758.427186:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760758.427199:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6373, 6373] 00000008:00010000:1.0:1789760758.427212:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760758.427216:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760758.427225:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26103808, 26107904) append 0 append lockpos 0 00000080:00200000:1.0:1789760758.427235:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26103808=0x18e5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760758.427268:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760758.427271:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26107904 00000020:00010000:1.0:1789760758.427276:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760758.427279:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.427282:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.427285:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760758.427287:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760758.427293:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760758.427307:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760758.427454:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:0.0:1789760758.427531:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514ad80 - object 0x240000400:5 offset 26103808<>26107904 00000100:00100000:0.0:1789760758.427552:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760758.427562:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512677504:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760758.505999:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973692 gen 3 00000008:00010000:0.0:1789760758.506045:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760758.506052:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760758.506061:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760758.506083:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760758.506105:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512677504:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760758.506200:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6373, 6373] 00000080:00200000:1.0:1789760758.506213:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760758.506217:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760758.506229:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26107904 i_blocks 51207 00000020:00200000:1.0:1789760758.506232:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760758.506239:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760758.506264:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26107904, count: 4096 00000080:00200000:1.0:1789760758.506270:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26107904, bytes: 4096 00000080:00200000:1.0:1789760758.506278:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760758.506285:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760758.506288:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26107904, 26112000) 0 0 00000080:00200000:1.0:1789760758.506291:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760758.506314:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6374, 6374] 00020000:00200000:1.0:1789760758.506321:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26107904 chunk: [26107904, 26112000) 26112000, 4096 00020000:00200000:1.0:1789760758.506324:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760758.506341:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6374, 6374] 00000008:00010000:1.0:1789760758.506353:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760758.506358:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760758.506383:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26107904, 26112000) append 0 append lockpos 0 00000080:00200000:1.0:1789760758.506394:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26107904=0x18e6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760758.506438:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760758.506441:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26112000 00000020:00010000:1.0:1789760758.506447:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760758.506449:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.506452:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.506455:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760758.506457:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760758.506463:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760758.506469:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760758.506482:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760758.506554:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97480 - object 0x240000400:5 offset 26107904<>26112000 00000100:00100000:1.0:1789760758.506591:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760758.506604:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512677632:192.168.203.156@tcp:4:dd.0 00010000:00010000:1.0:1789760758.557226:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:1.0:1789760758.560606:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760758.560617:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760758.560626:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760758.560645:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760758.560667:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512677632:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760758.560710:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6374, 6374] 00000080:00200000:3.0:1789760758.560722:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.560728:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.560739:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26112000 i_blocks 51207 00000020:00200000:3.0:1789760758.560742:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.560751:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.560775:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26112000, count: 4096 00000080:00200000:3.0:1789760758.560781:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26112000, bytes: 4096 00000080:00200000:3.0:1789760758.560789:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.560795:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.560797:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26112000, 26116096) 0 0 00000080:00200000:3.0:1789760758.560801:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.560813:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6375, 6375] 00020000:00200000:3.0:1789760758.560819:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26112000 chunk: [26112000, 26116096) 26116096, 4096 00020000:00200000:3.0:1789760758.560824:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.560839:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6375, 6375] 00000008:00010000:3.0:1789760758.560852:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.560857:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760758.560866:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26112000, 26116096) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.560876:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26112000=0x18e7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.560946:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.560948:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26116096 00000020:00010000:3.0:1789760758.560954:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.560957:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.560960:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.560962:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.560964:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.560970:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.560976:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.560990:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760758.561081:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 26112000<>26116096 00000100:00100000:2.0:1789760758.561145:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512677760:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760758.608696:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973694 gen 3 00000008:00010000:3.0:1789760758.608746:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:3.0:1789760758.608752:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760758.608760:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760758.608788:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:3.0:1789760758.608811:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512677760:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760758.610696:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6375, 6375] 00000080:00200000:1.0:1789760758.610714:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760758.610719:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760758.610732:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26116096 i_blocks 51207 00000020:00200000:1.0:1789760758.610736:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760758.610746:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760758.610771:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26116096, count: 4096 00000080:00200000:1.0:1789760758.611001:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26116096, bytes: 4096 00000080:00200000:1.0:1789760758.611013:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760758.611020:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760758.611022:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26116096, 26120192) 0 0 00000080:00200000:1.0:1789760758.611027:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760758.611039:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6376, 6376] 00020000:00200000:1.0:1789760758.611046:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26116096 chunk: [26116096, 26120192) 26120192, 4096 00020000:00200000:1.0:1789760758.611049:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760758.611065:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6376, 6376] 00000008:00010000:1.0:1789760758.611078:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760758.611082:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760758.611091:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26116096, 26120192) append 0 append lockpos 0 00000080:00200000:1.0:1789760758.611103:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26116096=0x18e8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760758.611149:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760758.611152:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26120192 00000020:00010000:1.0:1789760758.611159:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760758.611162:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.611165:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.611168:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760758.611170:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760758.611177:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760758.611182:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760758.611198:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760758.611278:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95880 - object 0x240000400:5 offset 26116096<>26120192 00000100:00100000:0.0:1789760758.611847:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512677888:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760758.650498:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973695 gen 3 00000008:00010000:0.0:1789760758.650534:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760758.650540:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760758.650548:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760758.650570:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760758.650593:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512677888:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760758.650748:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6376, 6376] 00000080:00200000:1.0:1789760758.650761:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760758.650767:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760758.650779:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26120192 i_blocks 51207 00000020:00200000:1.0:1789760758.650782:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760758.650800:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760758.650826:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26120192, count: 4096 00000080:00200000:1.0:1789760758.650831:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26120192, bytes: 4096 00000080:00200000:1.0:1789760758.650838:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760758.650845:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760758.651041:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26120192, 26124288) 0 0 00000080:00200000:1.0:1789760758.651046:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760758.651059:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6377, 6377] 00020000:00200000:1.0:1789760758.651067:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26120192 chunk: [26120192, 26124288) 26124288, 4096 00020000:00200000:1.0:1789760758.651070:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760758.651085:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6377, 6377] 00000008:00010000:1.0:1789760758.651100:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760758.651105:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760758.651113:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26120192, 26124288) append 0 append lockpos 0 00000080:00200000:1.0:1789760758.651124:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26120192=0x18e9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760758.651168:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760758.651171:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26124288 00000020:00010000:1.0:1789760758.651179:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760758.651182:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.651184:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760758.651187:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760758.651190:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760758.651196:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760758.651201:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760758.651215:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760758.651360:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96d80 - object 0x240000400:5 offset 26120192<>26124288 00000100:00100000:0.0:1789760758.651423:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512678016:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760758.712520:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973696 gen 3 00000008:00010000:0.0:1789760758.712556:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760758.712563:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760758.712571:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760758.712642:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760758.712672:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512678016:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760758.713011:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6377, 6377] 00000080:00200000:3.0:1789760758.713025:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.713030:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.713043:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26124288 i_blocks 51207 00000020:00200000:3.0:1789760758.713046:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.713118:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.713143:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26124288, count: 4096 00000080:00200000:3.0:1789760758.713150:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26124288, bytes: 4096 00000080:00200000:3.0:1789760758.713160:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.713167:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.713169:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26124288, 26128384) 0 0 00000080:00200000:3.0:1789760758.713173:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.713186:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6378, 6378] 00020000:00200000:3.0:1789760758.713193:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26124288 chunk: [26124288, 26128384) 26128384, 4096 00020000:00200000:3.0:1789760758.713196:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.713209:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6378, 6378] 00000008:00010000:3.0:1789760758.713222:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.713226:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760758.713235:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26124288, 26128384) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.713319:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26124288=0x18ea000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.713362:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.713365:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26128384 00000020:00010000:3.0:1789760758.713372:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.713375:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.713378:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.713382:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.713385:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.713391:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.713396:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.713410:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760758.714162:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 26124288<>26128384 00000100:00100000:2.0:1789760758.714213:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760758.714350:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512678144:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760758.765418:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:3.0:1789760758.765430:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760758.765439:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760758.765456:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:3.0:1789760758.765480:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512678144:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760758.765556:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6378, 6378] 00000080:00200000:2.0:1789760758.765570:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760758.765575:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760758.765587:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26128384 i_blocks 51207 00000020:00200000:2.0:1789760758.765589:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760758.765597:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760758.765623:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26128384, count: 4096 00000080:00200000:2.0:1789760758.765629:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26128384, bytes: 4096 00000080:00200000:2.0:1789760758.765637:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760758.765644:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760758.765646:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26128384, 26132480) 0 0 00000080:00200000:2.0:1789760758.765650:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760758.765663:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6379, 6379] 00020000:00200000:2.0:1789760758.765670:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26128384 chunk: [26128384, 26132480) 26132480, 4096 00020000:00200000:2.0:1789760758.765673:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760758.765700:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6379, 6379] 00000008:00010000:2.0:1789760758.765713:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760758.765719:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760758.765727:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26128384, 26132480) append 0 append lockpos 0 00000080:00200000:2.0:1789760758.765738:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26128384=0x18eb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760758.765779:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760758.765782:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26132480 00000020:00010000:2.0:1789760758.765797:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760758.765801:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760758.765804:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760758.765807:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760758.765810:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760758.765816:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760758.765821:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760758.765835:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760758.765952:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 26128384<>26132480 00000100:00100000:3.0:1789760758.766421:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760758.766437:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512678272:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760758.802655:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973698 gen 3 00000008:00010000:3.0:1789760758.802711:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760758.802718:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760758.802725:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760758.802743:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760758.802767:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512678272:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760758.803200:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6379, 6379] 00000080:00200000:2.0:1789760758.803214:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760758.803218:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760758.803230:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26132480 i_blocks 51207 00000020:00200000:2.0:1789760758.803233:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760758.803240:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760758.803266:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26132480, count: 4096 00000080:00200000:2.0:1789760758.803272:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26132480, bytes: 4096 00000080:00200000:2.0:1789760758.803279:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760758.803287:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760758.803289:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26132480, 26136576) 0 0 00000080:00200000:2.0:1789760758.803302:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760758.803316:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6380, 6380] 00020000:00200000:2.0:1789760758.803323:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26132480 chunk: [26132480, 26136576) 26136576, 4096 00020000:00200000:2.0:1789760758.803327:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760758.803342:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6380, 6380] 00000008:00010000:2.0:1789760758.803365:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760758.803371:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760758.803379:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26132480, 26136576) append 0 append lockpos 0 00000080:00200000:2.0:1789760758.803390:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26132480=0x18ec000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760758.803437:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760758.803439:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26136576 00000020:00010000:2.0:1789760758.803446:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760758.803450:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760758.803452:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760758.803456:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760758.803459:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760758.803465:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760758.803471:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760758.803485:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000008:00100000:2.0:1789760758.803562:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 26132480<>26136576 00000100:00100000:3.0:1789760758.803710:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512678400:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760758.837766:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973699 gen 3 00000008:00010000:3.0:1789760758.837851:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760758.837858:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760758.837866:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760758.837976:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760758.838000:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512678400:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760758.838205:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6380, 6380] 00000080:00200000:2.0:1789760758.838217:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760758.838223:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760758.838235:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26136576 i_blocks 51207 00000020:00200000:2.0:1789760758.838239:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760758.838248:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760758.838274:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26136576, count: 4096 00000080:00200000:2.0:1789760758.838280:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26136576, bytes: 4096 00000080:00200000:2.0:1789760758.838288:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760758.838306:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760758.838309:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26136576, 26140672) 0 0 00000080:00200000:2.0:1789760758.838312:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760758.838326:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6381, 6381] 00020000:00200000:2.0:1789760758.838332:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26136576 chunk: [26136576, 26140672) 26140672, 4096 00020000:00200000:2.0:1789760758.838336:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760758.838351:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6381, 6381] 00000008:00010000:2.0:1789760758.838364:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760758.838370:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760758.838380:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26136576, 26140672) append 0 append lockpos 0 00000080:00200000:2.0:1789760758.838391:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26136576=0x18ed000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760758.838434:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760758.838437:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26140672 00000020:00010000:2.0:1789760758.838443:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760758.838447:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760758.838450:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760758.838454:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760758.838456:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760758.838463:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760758.838469:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760758.838484:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000008:00100000:2.0:1789760758.838562:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 26136576<>26140672 00000100:00100000:3.0:1789760758.838717:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512678528:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760758.873946:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973700 gen 3 00000008:00010000:3.0:1789760758.873994:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760758.874000:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760758.874008:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760758.874024:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760758.874047:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512678528:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760758.874447:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6381, 6381] 00000080:00200000:2.0:1789760758.874460:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760758.874465:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760758.874479:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26140672 i_blocks 51207 00000020:00200000:2.0:1789760758.874483:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760758.874491:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760758.874517:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26140672, count: 4096 00000080:00200000:2.0:1789760758.874523:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26140672, bytes: 4096 00000080:00200000:2.0:1789760758.874531:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760758.874538:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760758.874540:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26140672, 26144768) 0 0 00000080:00200000:2.0:1789760758.874544:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760758.874556:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6382, 6382] 00020000:00200000:2.0:1789760758.874562:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26140672 chunk: [26140672, 26144768) 26144768, 4096 00020000:00200000:2.0:1789760758.874577:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760758.874592:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6382, 6382] 00000008:00010000:2.0:1789760758.874605:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760758.874610:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760758.874620:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26140672, 26144768) append 0 append lockpos 0 00000080:00200000:2.0:1789760758.874630:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26140672=0x18ee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760758.874675:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760758.874678:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26144768 00000020:00010000:2.0:1789760758.874800:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760758.874805:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760758.874808:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760758.874810:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760758.874813:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760758.874820:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760758.874826:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760758.874840:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760758.875170:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 26140672<>26144768 00000100:00100000:3.0:1789760758.875219:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760758.878805:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512678656:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760758.923733:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973701 gen 3 00000008:00010000:3.0:1789760758.923762:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760758.923767:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760758.923773:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760758.923819:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760758.923840:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512678656:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760758.923980:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6382, 6382] 00000080:00200000:0.0:1789760758.923992:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760758.923997:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760758.924009:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26144768 i_blocks 51207 00000020:00200000:0.0:1789760758.924012:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760758.924021:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760758.924046:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26144768, count: 4096 00000080:00200000:0.0:1789760758.924051:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26144768, bytes: 4096 00000080:00200000:0.0:1789760758.924060:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760758.924067:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760758.924069:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26144768, 26148864) 0 0 00000080:00200000:0.0:1789760758.924072:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760758.924084:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6383, 6383] 00020000:00200000:0.0:1789760758.924091:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26144768 chunk: [26144768, 26148864) 26148864, 4096 00020000:00200000:0.0:1789760758.924094:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760758.924108:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6383, 6383] 00000008:00010000:0.0:1789760758.924121:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760758.924126:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760758.924135:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26144768, 26148864) append 0 append lockpos 0 00000080:00200000:0.0:1789760758.924146:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26144768=0x18ef000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760758.924190:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760758.924192:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26148864 00000020:00010000:0.0:1789760758.924199:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760758.924202:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760758.924205:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760758.924208:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760758.924211:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760758.924217:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760758.924222:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760758.924235:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc600 00000008:00100000:0.0:1789760758.924325:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514ad80 - object 0x240000400:5 offset 26144768<>26148864 00000100:00100000:0.0:1789760758.924365:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760758.924377:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512678784:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760758.956354:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973702 gen 3 00000008:00010000:0.0:1789760758.956476:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:0.0:1789760758.956483:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760758.956492:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760758.956517:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:0.0:1789760758.956540:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512678784:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760758.956733:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6383, 6383] 00000080:00200000:3.0:1789760758.956745:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760758.956749:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760758.956759:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26148864 i_blocks 51207 00000020:00200000:3.0:1789760758.956762:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760758.956769:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760758.956792:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26148864, count: 4096 00000080:00200000:3.0:1789760758.956797:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26148864, bytes: 4096 00000080:00200000:3.0:1789760758.956805:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760758.956812:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760758.956814:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26148864, 26152960) 0 0 00000080:00200000:3.0:1789760758.956817:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760758.956830:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6384, 6384] 00020000:00200000:3.0:1789760758.956836:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26148864 chunk: [26148864, 26152960) 26152960, 4096 00020000:00200000:3.0:1789760758.956841:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760758.956857:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6384, 6384] 00000008:00010000:3.0:1789760758.956870:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760758.956875:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760758.956883:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26148864, 26152960) append 0 append lockpos 0 00000080:00200000:3.0:1789760758.956893:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26148864=0x18f0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760758.956975:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760758.956978:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26152960 00000020:00010000:3.0:1789760758.956984:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760758.956987:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.956990:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760758.956993:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760758.956995:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760758.957001:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760758.957006:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760758.957019:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760758.957091:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 26148864<>26152960 00000100:00100000:2.0:1789760758.957151:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512678912:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760758.990015:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973703 gen 3 00000008:00010000:3.0:1789760758.990050:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:3.0:1789760758.990056:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760758.990064:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760758.990085:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:3.0:1789760758.990105:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512678912:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760758.990307:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6384, 6384] 00000080:00200000:0.0:1789760758.990319:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760758.990324:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760758.990335:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26152960 i_blocks 51207 00000020:00200000:0.0:1789760758.990338:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760758.990346:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760758.990380:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26152960, count: 4096 00000080:00200000:0.0:1789760758.990387:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26152960, bytes: 4096 00000080:00200000:0.0:1789760758.990394:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760758.990401:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760758.990403:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26152960, 26157056) 0 0 00000080:00200000:0.0:1789760758.990407:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760758.990420:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6385, 6385] 00020000:00200000:0.0:1789760758.990427:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26152960 chunk: [26152960, 26157056) 26157056, 4096 00020000:00200000:0.0:1789760758.990430:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760758.990444:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6385, 6385] 00000008:00010000:0.0:1789760758.990457:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760758.990462:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760758.990471:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26152960, 26157056) append 0 append lockpos 0 00000080:00200000:0.0:1789760758.990482:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26152960=0x18f1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760758.990524:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760758.990526:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26157056 00000020:00010000:0.0:1789760758.990533:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760758.990537:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760758.990540:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760758.990542:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760758.990545:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760758.990551:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760758.990557:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760758.990571:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc600 00000008:00100000:0.0:1789760758.990651:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514ad80 - object 0x240000400:5 offset 26152960<>26157056 00000100:00100000:1.0:1789760758.990940:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760758.990955:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512679040:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760759.025654:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973704 gen 3 00000008:00010000:1.0:1789760759.025691:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760759.025697:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.025705:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.025772:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760759.025805:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512679040:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760759.026085:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6385, 6385] 00000080:00200000:0.0:1789760759.026098:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.026104:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.026192:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26157056 i_blocks 51207 00000020:00200000:0.0:1789760759.026197:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.026205:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.026232:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26157056, count: 4096 00000080:00200000:0.0:1789760759.026239:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26157056, bytes: 4096 00000080:00200000:0.0:1789760759.026247:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.026254:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.026256:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26157056, 26161152) 0 0 00000080:00200000:0.0:1789760759.026260:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.026273:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6386, 6386] 00020000:00200000:0.0:1789760759.026280:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26157056 chunk: [26157056, 26161152) 26161152, 4096 00020000:00200000:0.0:1789760759.026283:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.026306:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6386, 6386] 00000008:00010000:0.0:1789760759.026320:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.026325:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760759.026333:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26157056, 26161152) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.026345:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26157056=0x18f2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.026455:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.026458:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26161152 00000020:00010000:0.0:1789760759.026464:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.026468:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.026470:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.026473:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.026476:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.026483:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.026489:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.026501:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000008:00100000:0.0:1789760759.026573:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148700 - object 0x240000400:5 offset 26157056<>26161152 00000100:00100000:1.0:1789760759.026668:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512679168:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760759.055135:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973705 gen 3 00000008:00010000:1.0:1789760759.055181:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760759.055188:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.055200:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.055218:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760759.055240:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512679168:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760759.055418:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6386, 6386] 00000080:00200000:0.0:1789760759.055430:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.055435:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.055445:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26161152 i_blocks 51207 00000020:00200000:0.0:1789760759.055448:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.055456:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.055481:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26161152, count: 4096 00000080:00200000:0.0:1789760759.055487:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26161152, bytes: 4096 00000080:00200000:0.0:1789760759.055495:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.055502:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.055504:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26161152, 26165248) 0 0 00000080:00200000:0.0:1789760759.055507:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.055519:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6387, 6387] 00020000:00200000:0.0:1789760759.055525:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26161152 chunk: [26161152, 26165248) 26165248, 4096 00020000:00200000:0.0:1789760759.055528:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.055542:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6387, 6387] 00000008:00010000:0.0:1789760759.055555:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.055561:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760759.055569:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26161152, 26165248) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.055579:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26161152=0x18f3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.055622:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.055625:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26165248 00000020:00010000:0.0:1789760759.055630:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.055634:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.055636:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.055639:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.055641:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.055647:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.055652:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.055665:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cccc0 00000008:00100000:0.0:1789760759.055805:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514aa00 - object 0x240000400:5 offset 26161152<>26165248 00000100:00100000:1.0:1789760759.055858:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512679296:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760759.090455:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973706 gen 3 00000008:00010000:1.0:1789760759.090494:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760759.090501:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.090511:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.090532:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760759.090557:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512679296:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760759.090659:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6387, 6387] 00000080:00200000:0.0:1789760759.090671:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760759.090676:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760759.090688:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26165248 i_blocks 51207 00000020:00200000:0.0:1789760759.090692:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760759.090700:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760759.090723:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26165248, count: 4096 00000080:00200000:0.0:1789760759.090729:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26165248, bytes: 4096 00000080:00200000:0.0:1789760759.090737:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760759.090744:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760759.090746:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26165248, 26169344) 0 0 00000080:00200000:0.0:1789760759.090749:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760759.090762:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6388, 6388] 00020000:00200000:0.0:1789760759.090769:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26165248 chunk: [26165248, 26169344) 26169344, 4096 00020000:00200000:0.0:1789760759.090772:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760759.090786:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6388, 6388] 00000008:00010000:0.0:1789760759.090799:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760759.090805:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760759.090813:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26165248, 26169344) append 0 append lockpos 0 00000080:00200000:0.0:1789760759.090823:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26165248=0x18f4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760759.090866:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760759.090868:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26169344 00000020:00010000:0.0:1789760759.090875:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760759.090878:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.090881:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760759.090883:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760759.090886:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760759.090892:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760759.090897:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760759.090950:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc900 00000008:00100000:0.0:1789760759.091025:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514aa00 - object 0x240000400:5 offset 26165248<>26169344 00000100:00100000:0.0:1789760759.091055:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512679424:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760759.125077:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973707 gen 3 00000008:00010000:0.0:1789760759.125112:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:0.0:1789760759.125118:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760759.125127:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760759.125150:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:0.0:1789760759.125236:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512679424:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760759.125381:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6388, 6388] 00000080:00200000:1.0:1789760759.125393:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.125397:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.125408:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26169344 i_blocks 51207 00000020:00200000:1.0:1789760759.125411:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.125419:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.125444:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26169344, count: 4096 00000080:00200000:1.0:1789760759.125450:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26169344, bytes: 4096 00000080:00200000:1.0:1789760759.125459:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.125465:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.125467:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26169344, 26173440) 0 0 00000080:00200000:1.0:1789760759.125471:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.125483:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6389, 6389] 00020000:00200000:1.0:1789760759.125488:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26169344 chunk: [26169344, 26173440) 26173440, 4096 00020000:00200000:1.0:1789760759.125492:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.125508:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6389, 6389] 00000008:00010000:1.0:1789760759.125521:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.125525:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760759.125534:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26169344, 26173440) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.125544:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26169344=0x18f5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.125587:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.125589:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26173440 00000020:00010000:1.0:1789760759.125596:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.125599:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.125601:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.125604:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.125607:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.125613:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.125618:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.125632:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760759.125703:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96d80 - object 0x240000400:5 offset 26169344<>26173440 00000100:00100000:0.0:1789760759.125799:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760759.125811:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512679552:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760759.168738:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973708 gen 3 00000008:00010000:0.0:1789760759.168769:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760759.168773:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760759.168780:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760759.168801:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760759.168824:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512679552:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760759.168925:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6389, 6389] 00000080:00200000:1.0:1789760759.168937:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.168942:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.168953:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26173440 i_blocks 51207 00000020:00200000:1.0:1789760759.168957:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.168965:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.168989:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26173440, count: 4096 00000080:00200000:1.0:1789760759.168996:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26173440, bytes: 4096 00000080:00200000:1.0:1789760759.169003:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.169009:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.169011:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26173440, 26177536) 0 0 00000080:00200000:1.0:1789760759.169015:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.169029:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6390, 6390] 00020000:00200000:1.0:1789760759.169034:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26173440 chunk: [26173440, 26177536) 26177536, 4096 00020000:00200000:1.0:1789760759.169038:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.169052:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6390, 6390] 00000008:00010000:1.0:1789760759.169066:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.169070:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760759.169079:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26173440, 26177536) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.169089:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26173440=0x18f6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.169202:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.169205:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26177536 00000020:00010000:1.0:1789760759.169212:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.169215:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.169218:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.169221:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.169224:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.169230:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.169236:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.169249:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760759.169330:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95880 - object 0x240000400:5 offset 26173440<>26177536 00000100:00100000:0.0:1789760759.169380:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512679680:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760759.203412:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973709 gen 3 00000008:00010000:0.0:1789760759.203449:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760759.203455:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760759.203463:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760759.203483:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760759.203505:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512679680:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760759.203657:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6390, 6390] 00000080:00200000:1.0:1789760759.203668:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.203673:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.203684:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26177536 i_blocks 51207 00000020:00200000:1.0:1789760759.203687:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.203694:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.203725:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26177536, count: 4096 00000080:00200000:1.0:1789760759.203731:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26177536, bytes: 4096 00000080:00200000:1.0:1789760759.203738:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.203745:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.203747:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26177536, 26181632) 0 0 00000080:00200000:1.0:1789760759.203751:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.203763:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6391, 6391] 00020000:00200000:1.0:1789760759.203770:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26177536 chunk: [26177536, 26181632) 26181632, 4096 00020000:00200000:1.0:1789760759.203773:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.203787:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6391, 6391] 00000008:00010000:1.0:1789760759.203800:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.203804:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760759.203813:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26177536, 26181632) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.203823:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26177536=0x18f7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.203863:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.203866:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26181632 00000020:00010000:1.0:1789760759.203872:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.203875:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.203878:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.203881:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.203884:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.203890:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.203895:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.203956:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760759.204030:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97480 - object 0x240000400:5 offset 26177536<>26181632 00000100:00100000:0.0:1789760759.204123:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512679808:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760759.239206:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973710 gen 3 00000008:00010000:0.0:1789760759.239241:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760759.239247:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760759.239256:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760759.239275:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760759.239312:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512679808:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760759.239511:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6391, 6391] 00000080:00200000:1.0:1789760759.239523:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.239528:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.239539:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26181632 i_blocks 51207 00000020:00200000:1.0:1789760759.239543:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.239551:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.239576:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26181632, count: 4096 00000080:00200000:1.0:1789760759.239582:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26181632, bytes: 4096 00000080:00200000:1.0:1789760759.239589:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.239597:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.239599:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26181632, 26185728) 0 0 00000080:00200000:1.0:1789760759.239603:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.239617:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6392, 6392] 00020000:00200000:1.0:1789760759.239624:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26181632 chunk: [26181632, 26185728) 26185728, 4096 00020000:00200000:1.0:1789760759.239627:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.239645:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6392, 6392] 00000008:00010000:1.0:1789760759.239661:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.239665:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760759.239674:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26181632, 26185728) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.239695:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26181632=0x18f8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.239744:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.239747:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26185728 00000020:00010000:1.0:1789760759.239754:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.239757:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.239759:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.239763:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.239765:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.239772:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.239777:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.239791:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760759.239864:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96680 - object 0x240000400:5 offset 26181632<>26185728 00000100:00100000:0.0:1789760759.239970:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760759.239983:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512679936:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760759.283959:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760759.283980:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760759.283988:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760759.284007:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760759.284031:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512679936:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760759.284169:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6392, 6392] 00000080:00200000:1.0:1789760759.284182:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.284187:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.284198:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26185728 i_blocks 51207 00000020:00200000:1.0:1789760759.284201:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.284208:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.284233:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26185728, count: 4096 00000080:00200000:1.0:1789760759.284239:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26185728, bytes: 4096 00000080:00200000:1.0:1789760759.284246:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.284253:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.284255:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26185728, 26189824) 0 0 00000080:00200000:1.0:1789760759.284259:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.284272:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6393, 6393] 00020000:00200000:1.0:1789760759.284279:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26185728 chunk: [26185728, 26189824) 26189824, 4096 00020000:00200000:1.0:1789760759.284283:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.284309:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6393, 6393] 00000008:00010000:1.0:1789760759.284323:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.284328:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760759.284337:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26185728, 26189824) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.284347:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26185728=0x18f9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.284466:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.284470:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26189824 00000020:00010000:1.0:1789760759.284478:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.284481:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.284484:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.284487:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.284489:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.284495:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.284501:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.284514:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760759.284592:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 26185728<>26189824 00000100:00100000:0.0:1789760759.284870:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512680064:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760759.329881:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973712 gen 3 00000008:00010000:0.0:1789760759.329950:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760759.329956:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760759.329965:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760759.329988:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760759.330010:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512680064:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760759.330178:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6393, 6393] 00000080:00200000:1.0:1789760759.330190:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.330196:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.330206:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26189824 i_blocks 51207 00000020:00200000:1.0:1789760759.330210:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.330219:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.330244:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26189824, count: 4096 00000080:00200000:1.0:1789760759.330250:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26189824, bytes: 4096 00000080:00200000:1.0:1789760759.330256:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.330264:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.330266:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26189824, 26193920) 0 0 00000080:00200000:1.0:1789760759.330269:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.330281:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6394, 6394] 00020000:00200000:1.0:1789760759.330369:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26189824 chunk: [26189824, 26193920) 26193920, 4096 00020000:00200000:1.0:1789760759.330373:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.330389:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6394, 6394] 00000008:00010000:1.0:1789760759.330404:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.330409:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760759.330418:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26189824, 26193920) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.330428:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26189824=0x18fa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.330472:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.330475:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26193920 00000020:00010000:1.0:1789760759.330481:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.330484:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.330487:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.330490:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.330493:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.330499:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.330505:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.330517:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760759.330588:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 26189824<>26193920 00000100:00100000:1.0:1789760759.330626:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760759.330638:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512680192:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760759.379517:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973713 gen 3 00000008:00010000:1.0:1789760759.379554:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760759.379560:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.379568:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.379590:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760759.379614:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512680192:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760759.379760:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6394, 6394] 00000080:00200000:3.0:1789760759.379774:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760759.379779:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760759.379790:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26193920 i_blocks 51207 00000020:00200000:3.0:1789760759.379794:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760759.379801:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760759.379826:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26193920, count: 4096 00000080:00200000:3.0:1789760759.379832:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26193920, bytes: 4096 00000080:00200000:3.0:1789760759.379840:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760759.379848:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760759.379851:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26193920, 26198016) 0 0 00000080:00200000:3.0:1789760759.379856:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760759.379869:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6395, 6395] 00020000:00200000:3.0:1789760759.379876:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26193920 chunk: [26193920, 26198016) 26198016, 4096 00020000:00200000:3.0:1789760759.379880:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760759.379894:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6395, 6395] 00000008:00010000:3.0:1789760759.379934:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760759.379939:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760759.379947:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26193920, 26198016) append 0 append lockpos 0 00000080:00200000:3.0:1789760759.379957:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26193920=0x18fb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760759.380006:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760759.380009:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26198016 00000020:00010000:3.0:1789760759.380015:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760759.380018:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.380021:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.380023:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760759.380026:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760759.380032:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760759.380037:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.380211:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:1.0:1789760759.380306:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 26193920<>26198016 00000100:00100000:1.0:1789760759.380359:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512680320:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760759.420883:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973714 gen 3 00000008:00010000:1.0:1789760759.420929:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:1.0:1789760759.420935:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.420942:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.420957:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:1.0:1789760759.420974:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512680320:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760759.421137:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6395, 6395] 00000080:00200000:3.0:1789760759.421149:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760759.421154:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760759.421166:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26198016 i_blocks 51207 00000020:00200000:3.0:1789760759.421169:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760759.421177:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760759.421208:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26198016, count: 4096 00000080:00200000:3.0:1789760759.421215:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26198016, bytes: 4096 00000080:00200000:3.0:1789760759.421224:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760759.421231:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760759.421233:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26198016, 26202112) 0 0 00000080:00200000:3.0:1789760759.421237:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760759.421250:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6396, 6396] 00020000:00200000:3.0:1789760759.421256:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26198016 chunk: [26198016, 26202112) 26202112, 4096 00020000:00200000:3.0:1789760759.421260:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760759.421274:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6396, 6396] 00000008:00010000:3.0:1789760759.421288:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760759.421302:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760759.421311:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26198016, 26202112) append 0 append lockpos 0 00000080:00200000:3.0:1789760759.421321:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26198016=0x18fc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760759.421365:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760759.421368:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26202112 00000020:00010000:3.0:1789760759.421375:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760759.421378:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.421381:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.421384:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760759.421386:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760759.421391:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760759.421396:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760759.421410:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760759.421480:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 26198016<>26202112 00000100:00100000:2.0:1789760759.422032:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512680448:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760759.462430:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973715 gen 3 00000008:00010000:3.0:1789760759.462466:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:3.0:1789760759.462472:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760759.462482:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760759.462549:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:3.0:1789760759.462574:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512680448:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760759.462642:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6396, 6396] 00000080:00200000:1.0:1789760759.462655:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.462660:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.462672:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26202112 i_blocks 51207 00000020:00200000:1.0:1789760759.462675:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.462747:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.462771:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26202112, count: 4096 00000080:00200000:1.0:1789760759.462777:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26202112, bytes: 4096 00000080:00200000:1.0:1789760759.462785:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.462853:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.462855:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26202112, 26206208) 0 0 00000080:00200000:1.0:1789760759.462858:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.462873:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6397, 6397] 00020000:00200000:1.0:1789760759.462880:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26202112 chunk: [26202112, 26206208) 26206208, 4096 00020000:00200000:1.0:1789760759.462883:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.463047:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6397, 6397] 00000008:00010000:1.0:1789760759.463063:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.463067:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760759.463076:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26202112, 26206208) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.463087:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26202112=0x18fd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.463130:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.463133:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26206208 00000020:00010000:1.0:1789760759.463139:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.463142:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.463145:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.463148:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.463151:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.463157:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.463163:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.463177:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760759.463259:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 26202112<>26206208 00000100:00100000:0.0:1789760759.463345:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512680576:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760759.494666:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973716 gen 3 00000008:00010000:0.0:1789760759.494703:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760759.494710:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760759.494718:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760759.494738:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000080:00200000:1.0:1789760759.494929:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6397, 6397] 00000080:00200000:1.0:1789760759.494944:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.494949:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.494960:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26206208 i_blocks 51207 00000020:00200000:1.0:1789760759.494972:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.494980:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.495005:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26206208, count: 4096 00000080:00200000:1.0:1789760759.495011:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26206208, bytes: 4096 00000080:00200000:1.0:1789760759.495019:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.495026:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.495028:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26206208, 26210304) 0 0 00000080:00200000:1.0:1789760759.495032:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.495045:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6398, 6398] 00020000:00200000:1.0:1789760759.495052:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26206208 chunk: [26206208, 26210304) 26210304, 4096 00020000:00200000:1.0:1789760759.495055:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.495069:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6398, 6398] 00000008:00010000:1.0:1789760759.495081:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.495086:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760759.495094:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26206208, 26210304) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.495104:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26206208=0x18fe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.495140:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.495143:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26210304 00000020:00010000:1.0:1789760759.495149:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.495152:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.495155:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.495159:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.495162:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.495168:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.495173:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1789760759.495286:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512680576:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760759.495328:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:0.0:1789760759.495402:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514aa00 - object 0x240000400:5 offset 26206208<>26210304 00000100:00100000:0.0:1789760759.495426:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760759.495437:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512680704:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760759.523149:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973717 gen 3 00000008:00010000:0.0:1789760759.523184:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760759.523191:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760759.523199:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760759.523219:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760759.523243:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512680704:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760759.523347:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6398, 6398] 00000080:00200000:1.0:1789760759.523360:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.523365:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.523376:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26210304 i_blocks 51207 00000020:00200000:1.0:1789760759.523379:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.523387:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.523415:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26210304, count: 4096 00000080:00200000:1.0:1789760759.523421:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26210304, bytes: 4096 00000080:00200000:1.0:1789760759.523428:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.523435:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.523437:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26210304, 26214400) 0 0 00000080:00200000:1.0:1789760759.523441:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.523453:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6399, 6399] 00020000:00200000:1.0:1789760759.523459:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26210304 chunk: [26210304, 26214400) 26214400, 4096 00020000:00200000:1.0:1789760759.523463:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.523476:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6399, 6399] 00000008:00010000:1.0:1789760759.523488:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.523492:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760759.523501:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26210304, 26214400) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.523511:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26210304=0x18ff000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.523554:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.523557:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26214400 00000020:00010000:1.0:1789760759.523563:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.523567:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.523570:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.523573:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.523576:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.523583:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.523588:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.523601:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760759.523678:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 26210304<>26214400 00000100:00100000:1.0:1789760759.523723:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760759.523736:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512680832:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760759.554406:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973718 gen 3 00000008:00010000:1.0:1789760759.554443:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760759.554449:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760759.554457:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760759.554472:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760759.554494:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512680832:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760759.554573:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6399, 6399] 00000080:00200000:3.0:1789760759.554587:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760759.554592:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760759.554603:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26214400 i_blocks 51207 00000020:00200000:3.0:1789760759.554606:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760759.554613:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760759.554641:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26214400, count: 4096 00000080:00200000:3.0:1789760759.554647:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26214400, bytes: 4096 00000080:00200000:3.0:1789760759.554655:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760759.554663:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760759.554665:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26214400, 26218496) 0 0 00000080:00200000:3.0:1789760759.554669:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760759.554683:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6400, 6400] 00020000:00200000:3.0:1789760759.554689:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26214400 chunk: [26214400, 26218496) 26218496, 4096 00020000:00200000:3.0:1789760759.554693:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760759.554708:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6400, 6400] 00000008:00010000:3.0:1789760759.554722:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760759.554726:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760759.554735:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26214400, 26218496) append 0 append lockpos 0 00000080:00200000:3.0:1789760759.554745:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26214400=0x1900000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760759.554800:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760759.554802:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26218496 00000020:00010000:3.0:1789760759.554809:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760759.554812:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.554815:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760759.554818:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760759.554820:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760759.554826:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760759.554831:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760759.554845:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760759.554971:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 26214400<>26218496 00000100:00100000:3.0:1789760759.555010:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760759.555023:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512680960:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760759.581178:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512681216:192.168.203.156@tcp:400:kworker.0 00010000:00010000:0.0:1789760759.581550:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760759.581705:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1789760759.581713:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e6cf7267-9b19-462a-a263-14115df15dab:2360:1876699512681088:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760759.581752:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2360:1876699512681344:192.168.203.156@tcp:400:kworker.0 00000100:00100000:2.0:1789760759.583373:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e6cf7267-9b19-462a-a263-14115df15dab:2360:1876699512681088:192.168.203.156@tcp:400:kworker.0 00000100:00100000:2.0:1789760759.583386:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2360:1876699512681344:192.168.203.156@tcp:400:kworker.0 00000100:00100000:2.0:1789760759.583392:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512681216:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760759.585747:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973719 gen 3 00000008:00010000:3.0:1789760759.585780:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:3.0:1789760759.585786:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760759.585794:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760759.585810:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:3.0:1789760759.585830:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512680960:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760759.585926:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6400, 6400] 00000080:00200000:2.0:1789760759.585940:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760759.585945:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760759.585956:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26218496 i_blocks 51207 00000020:00200000:2.0:1789760759.585967:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760759.585976:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760759.586004:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26218496, count: 4096 00000080:00200000:2.0:1789760759.586010:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26218496, bytes: 4096 00000080:00200000:2.0:1789760759.586018:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760759.586025:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760759.586027:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26218496, 26222592) 0 0 00000080:00200000:2.0:1789760759.586031:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760759.586044:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6401, 6401] 00020000:00200000:2.0:1789760759.586050:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26218496 chunk: [26218496, 26222592) 26222592, 4096 00020000:00200000:2.0:1789760759.586054:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760759.586069:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6401, 6401] 00000008:00010000:2.0:1789760759.586081:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760759.586085:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760759.586094:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26218496, 26222592) append 0 append lockpos 0 00000080:00200000:2.0:1789760759.586104:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26218496=0x1901000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760759.586143:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760759.586145:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26222592 00000020:00010000:2.0:1789760759.586151:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760759.586154:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.586157:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.586159:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760759.586162:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760759.586168:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760759.586173:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760759.586186:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760759.586257:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 26218496<>26222592 00000100:00100000:2.0:1789760759.586303:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760759.586315:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512681472:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760759.611734:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973720 gen 3 00000008:00010000:3.0:1789760759.611771:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760759.611777:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760759.611786:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760759.611801:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760759.611824:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512681472:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760759.611930:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6401, 6401] 00000080:00200000:2.0:1789760759.611943:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760759.611948:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760759.611967:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26222592 i_blocks 53255 00000020:00200000:2.0:1789760759.611971:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760759.611979:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760759.612004:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26222592, count: 4096 00000080:00200000:2.0:1789760759.612011:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26222592, bytes: 4096 00000080:00200000:2.0:1789760759.612018:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760759.612025:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760759.612027:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26222592, 26226688) 0 0 00000080:00200000:2.0:1789760759.612030:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760759.612043:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6402, 6402] 00020000:00200000:2.0:1789760759.612049:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26222592 chunk: [26222592, 26226688) 26226688, 4096 00020000:00200000:2.0:1789760759.612053:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760759.612069:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6402, 6402] 00000008:00010000:2.0:1789760759.612083:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760759.612088:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760759.612097:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26222592, 26226688) append 0 append lockpos 0 00000080:00200000:2.0:1789760759.612107:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26222592=0x1902000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760759.612149:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760759.612152:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26226688 00000020:00010000:2.0:1789760759.612158:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760759.612162:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.612164:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.612167:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760759.612169:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760759.612175:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760759.612180:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760759.612192:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000008:00100000:2.0:1789760759.612266:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 26222592<>26226688 00000100:00100000:3.0:1789760759.612360:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512681600:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760759.645397:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973721 gen 3 00000008:00010000:3.0:1789760759.645434:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760759.645441:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760759.645449:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760759.645467:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760759.645487:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512681600:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760759.645572:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6402, 6402] 00000080:00200000:2.0:1789760759.645583:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760759.645588:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760759.645599:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26226688 i_blocks 53255 00000020:00200000:2.0:1789760759.645602:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760759.645609:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760759.645634:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26226688, count: 4096 00000080:00200000:2.0:1789760759.645640:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26226688, bytes: 4096 00000080:00200000:2.0:1789760759.645647:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760759.645653:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760759.645665:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26226688, 26230784) 0 0 00000080:00200000:2.0:1789760759.645669:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760759.645681:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6403, 6403] 00020000:00200000:2.0:1789760759.645687:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26226688 chunk: [26226688, 26230784) 26230784, 4096 00020000:00200000:2.0:1789760759.645691:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760759.645707:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6403, 6403] 00000008:00010000:2.0:1789760759.645721:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760759.645730:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760759.645740:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26226688, 26230784) append 0 append lockpos 0 00000080:00200000:2.0:1789760759.645752:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26226688=0x1903000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760759.645797:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760759.645799:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26230784 00000020:00010000:2.0:1789760759.645806:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760759.645809:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.645812:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760759.645815:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760759.645819:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760759.645825:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760759.645831:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760759.645851:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760759.646686:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 26226688<>26230784 00000100:00100000:3.0:1789760759.646755:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512681728:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760759.686853:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760759.686865:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760759.687093:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760759.687160:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760759.687186:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512681728:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760759.687244:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6403, 6403] 00000080:00200000:1.0:1789760759.687255:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.687260:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.687272:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26230784 i_blocks 53255 00000020:00200000:1.0:1789760759.687507:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.687518:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.687544:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26230784, count: 4096 00000080:00200000:1.0:1789760759.687551:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26230784, bytes: 4096 00000080:00200000:1.0:1789760759.687560:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.687567:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.687569:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26230784, 26234880) 0 0 00000080:00200000:1.0:1789760759.687572:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.687586:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6404, 6404] 00020000:00200000:1.0:1789760759.687594:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26230784 chunk: [26230784, 26234880) 26234880, 4096 00020000:00200000:1.0:1789760759.687598:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.687616:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6404, 6404] 00000008:00010000:1.0:1789760759.687630:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.687635:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760759.687645:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26230784, 26234880) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.687656:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26230784=0x1904000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.687702:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.687705:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26234880 00000020:00010000:1.0:1789760759.687712:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.687715:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.687717:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.687720:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.687722:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.687729:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.687735:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.687750:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760759.687838:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 26230784<>26234880 00000100:00100000:0.0:1789760759.688122:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512681856:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760759.739110:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973723 gen 3 00000008:00010000:0.0:1789760759.739163:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760759.739170:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760759.739179:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760759.739202:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760759.739227:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512681856:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760759.740425:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6404, 6404] 00000080:00200000:1.0:1789760759.740441:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.740446:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.740460:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26234880 i_blocks 53255 00000020:00200000:1.0:1789760759.740464:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.740474:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.740501:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26234880, count: 4096 00000080:00200000:1.0:1789760759.740507:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26234880, bytes: 4096 00000080:00200000:1.0:1789760759.740517:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.740524:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.740526:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26234880, 26238976) 0 0 00000080:00200000:1.0:1789760759.740530:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.740544:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6405, 6405] 00020000:00200000:1.0:1789760759.740551:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26234880 chunk: [26234880, 26238976) 26238976, 4096 00020000:00200000:1.0:1789760759.740555:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.740570:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6405, 6405] 00000008:00010000:1.0:1789760759.740583:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.740587:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760759.740596:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26234880, 26238976) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.740608:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26234880=0x1905000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.740656:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.740659:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26238976 00000020:00010000:1.0:1789760759.740665:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.740668:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.740671:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.740674:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.740676:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.740683:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.740688:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.740703:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760759.740781:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94380 - object 0x240000400:5 offset 26234880<>26238976 00000100:00100000:0.0:1789760759.742451:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512681984:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760759.785140:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973724 gen 3 00000008:00010000:0.0:1789760759.785175:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760759.785180:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760759.785188:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760759.785248:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760759.785272:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512681984:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760759.786080:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6405, 6405] 00000080:00200000:1.0:1789760759.786094:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.786099:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.786112:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26238976 i_blocks 53255 00000020:00200000:1.0:1789760759.786116:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.786125:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.786332:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26238976, count: 4096 00000080:00200000:1.0:1789760759.786339:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26238976, bytes: 4096 00000080:00200000:1.0:1789760759.786484:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.786492:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.786494:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26238976, 26243072) 0 0 00000080:00200000:1.0:1789760759.786498:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.786510:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6406, 6406] 00020000:00200000:1.0:1789760759.786517:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26238976 chunk: [26238976, 26243072) 26243072, 4096 00020000:00200000:1.0:1789760759.786521:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.786538:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6406, 6406] 00000008:00010000:1.0:1789760759.786552:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.786556:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760759.786565:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26238976, 26243072) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.786576:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26238976=0x1906000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.786625:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.786628:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26243072 00000020:00010000:1.0:1789760759.786634:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.786638:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.786640:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.786644:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.786646:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.786653:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.786658:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.786679:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760759.786751:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 26238976<>26243072 00000100:00100000:0.0:1789760759.787257:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512682112:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760759.838705:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973725 gen 3 00000008:00010000:0.0:1789760759.838737:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760759.838742:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760759.838750:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760759.838769:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760759.838790:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512682112:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760759.840518:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6406, 6406] 00000080:00200000:1.0:1789760759.840534:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.840539:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.840551:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26243072 i_blocks 53255 00000020:00200000:1.0:1789760759.840555:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.840567:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.840591:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26243072, count: 4096 00000080:00200000:1.0:1789760759.840598:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26243072, bytes: 4096 00000080:00200000:1.0:1789760759.840607:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.840614:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.840616:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26243072, 26247168) 0 0 00000080:00200000:1.0:1789760759.840620:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.840634:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6407, 6407] 00020000:00200000:1.0:1789760759.840640:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26243072 chunk: [26243072, 26247168) 26247168, 4096 00020000:00200000:1.0:1789760759.840644:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.840660:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6407, 6407] 00000008:00010000:1.0:1789760759.840673:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.840677:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760759.840686:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26243072, 26247168) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.840697:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26243072=0x1907000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.840746:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.840749:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26247168 00000020:00010000:1.0:1789760759.840756:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.840759:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.840762:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.840765:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.840767:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.840774:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.840779:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.840794:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760759.840880:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94700 - object 0x240000400:5 offset 26243072<>26247168 00000100:00100000:0.0:1789760759.843557:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760759.843586:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512682240:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760759.886536:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973726 gen 3 00000008:00010000:0.0:1789760759.886572:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760759.886579:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760759.886587:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760759.886609:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760759.886631:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512682240:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760759.886712:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6407, 6407] 00000080:00200000:1.0:1789760759.886724:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.886730:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.886741:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26247168 i_blocks 53255 00000020:00200000:1.0:1789760759.886745:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.886753:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.886779:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26247168, count: 4096 00000080:00200000:1.0:1789760759.886786:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26247168, bytes: 4096 00000080:00200000:1.0:1789760759.886793:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.886800:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.886802:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26247168, 26251264) 0 0 00000080:00200000:1.0:1789760759.886806:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.886819:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6408, 6408] 00020000:00200000:1.0:1789760759.886825:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26247168 chunk: [26247168, 26251264) 26251264, 4096 00020000:00200000:1.0:1789760759.886829:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.886844:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6408, 6408] 00000008:00010000:1.0:1789760759.886864:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.886868:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760759.886876:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26247168, 26251264) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.886887:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26247168=0x1908000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.886952:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.886955:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26251264 00000020:00010000:1.0:1789760759.886961:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.886964:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.886966:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.886969:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.886972:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.886978:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.886983:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.886997:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760759.887069:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 26247168<>26251264 00000100:00100000:0.0:1789760759.887265:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512682368:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760759.920600:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760759.920611:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760759.920621:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760759.920644:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760759.920668:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512682368:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760759.920771:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6408, 6408] 00000080:00200000:1.0:1789760759.920784:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.920789:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.920800:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26251264 i_blocks 53255 00000020:00200000:1.0:1789760759.920804:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.920811:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.920837:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26251264, count: 4096 00000080:00200000:1.0:1789760759.920843:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26251264, bytes: 4096 00000080:00200000:1.0:1789760759.920851:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.920857:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.920859:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26251264, 26255360) 0 0 00000080:00200000:1.0:1789760759.920863:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.920877:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6409, 6409] 00020000:00200000:1.0:1789760759.920883:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26251264 chunk: [26251264, 26255360) 26255360, 4096 00020000:00200000:1.0:1789760759.920887:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.920939:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6409, 6409] 00000008:00010000:1.0:1789760759.920952:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.920956:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760759.920966:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26251264, 26255360) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.920979:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26251264=0x1909000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.921021:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.921024:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26255360 00000020:00010000:1.0:1789760759.921031:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.921033:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.921036:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.921039:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.921052:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.921059:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.921064:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.921079:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760759.921155:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 26251264<>26255360 00000100:00100000:0.0:1789760759.921545:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760759.921559:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512682496:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760759.954930:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973728 gen 3 00000008:00010000:0.0:1789760759.954975:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760759.954981:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760759.954989:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760759.955013:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760759.955037:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512682496:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760759.955129:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6409, 6409] 00000080:00200000:1.0:1789760759.955141:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760759.955146:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760759.955157:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26255360 i_blocks 53255 00000020:00200000:1.0:1789760759.955160:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760759.955168:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760759.955193:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26255360, count: 4096 00000080:00200000:1.0:1789760759.955199:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26255360, bytes: 4096 00000080:00200000:1.0:1789760759.955206:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760759.955214:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760759.955216:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26255360, 26259456) 0 0 00000080:00200000:1.0:1789760759.955221:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760759.955234:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6410, 6410] 00020000:00200000:1.0:1789760759.955240:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26255360 chunk: [26255360, 26259456) 26259456, 4096 00020000:00200000:1.0:1789760759.955243:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760759.955258:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6410, 6410] 00000008:00010000:1.0:1789760759.955270:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760759.955274:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760759.955283:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26255360, 26259456) append 0 append lockpos 0 00000080:00200000:1.0:1789760759.955307:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26255360=0x190a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760759.955350:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760759.955352:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26259456 00000020:00010000:1.0:1789760759.955359:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760759.955361:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.955364:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760759.955367:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760759.955378:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760759.955384:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760759.955390:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760759.955403:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760759.955474:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95c00 - object 0x240000400:5 offset 26255360<>26259456 00000100:00100000:0.0:1789760759.955533:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512682624:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760760.001752:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973729 gen 3 00000008:00010000:0.0:1789760760.001786:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760760.001793:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760760.001801:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760760.001821:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760760.001850:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512682624:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760760.001996:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6410, 6410] 00000080:00200000:1.0:1789760760.002008:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760760.002013:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760760.002023:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26259456 i_blocks 53255 00000020:00200000:1.0:1789760760.002027:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760760.002034:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760760.002058:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26259456, count: 4096 00000080:00200000:1.0:1789760760.002064:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26259456, bytes: 4096 00000080:00200000:1.0:1789760760.002071:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760760.002078:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760760.002080:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26259456, 26263552) 0 0 00000080:00200000:1.0:1789760760.002084:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760760.002096:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6411, 6411] 00020000:00200000:1.0:1789760760.002103:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26259456 chunk: [26259456, 26263552) 26263552, 4096 00020000:00200000:1.0:1789760760.002107:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760760.002120:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6411, 6411] 00000008:00010000:1.0:1789760760.002132:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760760.002137:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760760.002145:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26259456, 26263552) append 0 append lockpos 0 00000080:00200000:1.0:1789760760.002157:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26259456=0x190b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760760.002197:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760760.002200:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26263552 00000020:00010000:1.0:1789760760.002207:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760760.002210:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.002213:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.002216:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760760.002219:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760760.002225:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760760.002230:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760760.002243:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760760.002325:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 26259456<>26263552 00000100:00100000:0.0:1789760760.002390:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512682752:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760760.041449:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760760.041461:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760760.041470:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760760.041488:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760760.041512:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512682752:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760760.041597:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6411, 6411] 00000080:00200000:1.0:1789760760.041608:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760760.041614:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760760.041626:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26263552 i_blocks 53255 00000020:00200000:1.0:1789760760.041629:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760760.041638:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760760.041664:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26263552, count: 4096 00000080:00200000:1.0:1789760760.041670:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26263552, bytes: 4096 00000080:00200000:1.0:1789760760.041677:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760760.041684:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760760.041686:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26263552, 26267648) 0 0 00000080:00200000:1.0:1789760760.041690:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760760.041702:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6412, 6412] 00020000:00200000:1.0:1789760760.041709:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26263552 chunk: [26263552, 26267648) 26267648, 4096 00020000:00200000:1.0:1789760760.041713:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760760.041728:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6412, 6412] 00000008:00010000:1.0:1789760760.041741:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760760.041745:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760760.041754:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26263552, 26267648) append 0 append lockpos 0 00000080:00200000:1.0:1789760760.041765:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26263552=0x190c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760760.041809:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760760.041812:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26267648 00000020:00010000:1.0:1789760760.041819:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760760.041822:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.041825:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.041828:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760760.041830:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760760.041836:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760760.041842:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760760.041856:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760760.041965:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 26263552<>26267648 00000100:00100000:0.0:1789760760.043382:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512682880:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760760.098646:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973731 gen 3 00000008:00010000:0.0:1789760760.098757:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760760.098764:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760760.098772:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760760.098793:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760760.098811:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512682880:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760760.099044:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6412, 6412] 00000080:00200000:1.0:1789760760.099056:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760760.099060:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760760.099071:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26267648 i_blocks 53255 00000020:00200000:1.0:1789760760.099074:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760760.099082:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760760.099108:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26267648, count: 4096 00000080:00200000:1.0:1789760760.099114:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26267648, bytes: 4096 00000080:00200000:1.0:1789760760.099125:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760760.099132:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760760.099134:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26267648, 26271744) 0 0 00000080:00200000:1.0:1789760760.099138:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760760.099151:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6413, 6413] 00020000:00200000:1.0:1789760760.099158:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26267648 chunk: [26267648, 26271744) 26271744, 4096 00020000:00200000:1.0:1789760760.099161:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760760.099177:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6413, 6413] 00000008:00010000:1.0:1789760760.099192:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760760.099197:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760760.099205:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26267648, 26271744) append 0 append lockpos 0 00000080:00200000:1.0:1789760760.099217:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26267648=0x190d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760760.099259:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760760.099262:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26271744 00000020:00010000:1.0:1789760760.099269:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760760.099272:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.099274:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.099277:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760760.099279:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760760.099296:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760760.099302:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760760.099315:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760760.099403:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97100 - object 0x240000400:5 offset 26267648<>26271744 00000100:00100000:0.0:1789760760.099557:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512683008:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760760.135850:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973732 gen 3 00000008:00010000:0.0:1789760760.135885:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760760.135891:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760760.135931:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760760.135943:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760760.135971:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512683008:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760760.136074:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6413, 6413] 00000080:00200000:1.0:1789760760.136086:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760760.136091:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760760.136104:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26271744 i_blocks 53255 00000020:00200000:1.0:1789760760.136108:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760760.136117:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760760.136146:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26271744, count: 4096 00000080:00200000:1.0:1789760760.136153:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26271744, bytes: 4096 00000080:00200000:1.0:1789760760.136161:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760760.136168:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760760.136170:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26271744, 26275840) 0 0 00000080:00200000:1.0:1789760760.136174:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760760.136189:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6414, 6414] 00020000:00200000:1.0:1789760760.136198:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26271744 chunk: [26271744, 26275840) 26275840, 4096 00020000:00200000:1.0:1789760760.136202:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760760.136222:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6414, 6414] 00000008:00010000:1.0:1789760760.136238:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760760.136242:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760760.136251:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26271744, 26275840) append 0 append lockpos 0 00000080:00200000:1.0:1789760760.136263:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26271744=0x190e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760760.136320:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760760.136323:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26275840 00000020:00010000:1.0:1789760760.136329:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760760.136332:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.136335:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.136337:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760760.136340:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760760.136345:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760760.136350:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760760.136364:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760760.136437:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 26271744<>26275840 00000100:00100000:0.0:1789760760.136493:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512683136:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760760.196618:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973733 gen 3 00000008:00010000:0.0:1789760760.196650:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760760.196656:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760760.196673:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760760.196695:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760760.196717:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512683136:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760760.196953:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6414, 6414] 00000080:00200000:1.0:1789760760.196965:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760760.196970:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760760.196982:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26275840 i_blocks 53255 00000020:00200000:1.0:1789760760.196986:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760760.196994:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760760.197016:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26275840, count: 4096 00000080:00200000:1.0:1789760760.197023:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26275840, bytes: 4096 00000080:00200000:1.0:1789760760.197031:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760760.197039:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760760.197041:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26275840, 26279936) 0 0 00000080:00200000:1.0:1789760760.197045:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760760.197058:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6415, 6415] 00020000:00200000:1.0:1789760760.197064:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26275840 chunk: [26275840, 26279936) 26279936, 4096 00020000:00200000:1.0:1789760760.197067:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760760.197081:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6415, 6415] 00000008:00010000:1.0:1789760760.197094:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760760.197098:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760760.197107:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26275840, 26279936) append 0 append lockpos 0 00000080:00200000:1.0:1789760760.197118:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26275840=0x190f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760760.197160:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760760.197163:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26279936 00000020:00010000:1.0:1789760760.197169:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760760.197172:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.197175:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760760.197178:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760760.197181:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760760.197187:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760760.197192:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760760.197204:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760760.198111:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 26275840<>26279936 00000100:00100000:0.0:1789760760.198164:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760760.201142:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512683264:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760760.267226:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973734 gen 3 00000008:00010000:0.0:1789760760.267260:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760760.267266:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760760.267274:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760760.267302:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760760.267333:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512683264:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760760.267569:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6415, 6415] 00000080:00200000:3.0:1789760760.267582:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760760.267587:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760760.267599:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26279936 i_blocks 53255 00000020:00200000:3.0:1789760760.267602:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760760.267610:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760760.267633:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26279936, count: 4096 00000080:00200000:3.0:1789760760.267639:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26279936, bytes: 4096 00000080:00200000:3.0:1789760760.267646:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760760.267652:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760760.267654:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26279936, 26284032) 0 0 00000080:00200000:3.0:1789760760.267658:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760760.267670:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6416, 6416] 00020000:00200000:3.0:1789760760.267688:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26279936 chunk: [26279936, 26284032) 26284032, 4096 00020000:00200000:3.0:1789760760.267691:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760760.267706:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6416, 6416] 00000008:00010000:3.0:1789760760.267721:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760760.267726:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760760.267736:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26279936, 26284032) append 0 append lockpos 0 00000080:00200000:3.0:1789760760.267749:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26279936=0x1910000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760760.267795:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760760.267798:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26284032 00000020:00010000:3.0:1789760760.267804:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760760.267808:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760760.267811:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760760.267814:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760760.267816:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760760.267823:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760760.267828:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760760.267843:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760760.267959:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 26279936<>26284032 00000100:00100000:3.0:1789760760.267996:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760760.268009:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512683392:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760760.313430:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973735 gen 3 00000008:00010000:3.0:1789760760.313466:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:3.0:1789760760.313473:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760760.313482:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760760.313504:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:3.0:1789760760.313527:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512683392:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760760.313627:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6416, 6416] 00000080:00200000:0.0:1789760760.313642:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760760.313646:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760760.313658:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26284032 i_blocks 53255 00000020:00200000:0.0:1789760760.313662:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760760.313669:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760760.313706:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26284032, count: 4096 00000080:00200000:0.0:1789760760.313712:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26284032, bytes: 4096 00000080:00200000:0.0:1789760760.313721:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760760.313728:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760760.313730:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26284032, 26288128) 0 0 00000080:00200000:0.0:1789760760.313734:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760760.313748:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6417, 6417] 00020000:00200000:0.0:1789760760.313755:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26284032 chunk: [26284032, 26288128) 26288128, 4096 00020000:00200000:0.0:1789760760.313759:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760760.313774:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6417, 6417] 00000008:00010000:0.0:1789760760.313788:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760760.313793:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760760.313801:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26284032, 26288128) append 0 append lockpos 0 00000080:00200000:0.0:1789760760.313820:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26284032=0x1911000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760760.313864:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760760.313867:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26288128 00000020:00010000:0.0:1789760760.313874:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760760.313878:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.313881:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.313884:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760760.313887:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760760.313893:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760760.313935:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760760.313950:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc900 00000008:00100000:0.0:1789760760.314031:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514aa00 - object 0x240000400:5 offset 26284032<>26288128 00000100:00100000:1.0:1789760760.314178:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512683520:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760760.352177:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973736 gen 3 00000008:00010000:1.0:1789760760.352232:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760760.352238:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760760.352247:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760760.352271:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760760.352304:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512683520:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760760.354112:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6417, 6417] 00000080:00200000:0.0:1789760760.354131:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760760.354136:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760760.354149:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26288128 i_blocks 53255 00000020:00200000:0.0:1789760760.354153:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760760.354164:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760760.354189:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26288128, count: 4096 00000080:00200000:0.0:1789760760.354196:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26288128, bytes: 4096 00000080:00200000:0.0:1789760760.354207:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760760.354217:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760760.354219:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26288128, 26292224) 0 0 00000080:00200000:0.0:1789760760.354222:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760760.354235:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6418, 6418] 00020000:00200000:0.0:1789760760.354242:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26288128 chunk: [26288128, 26292224) 26292224, 4096 00020000:00200000:0.0:1789760760.354246:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760760.354263:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6418, 6418] 00000008:00010000:0.0:1789760760.354278:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760760.354291:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760760.354300:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26288128, 26292224) append 0 append lockpos 0 00000080:00200000:0.0:1789760760.354311:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26288128=0x1912000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760760.354358:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760760.354360:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26292224 00000020:00010000:0.0:1789760760.354367:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760760.354370:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.354372:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.354375:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760760.354378:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760760.354384:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760760.354389:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760760.354406:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000008:00100000:0.0:1789760760.354487:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148700 - object 0x240000400:5 offset 26288128<>26292224 00000100:00100000:1.0:1789760760.354602:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512683648:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760760.423209:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973737 gen 3 00000008:00010000:1.0:1789760760.423245:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760760.423252:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760760.423261:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760760.423291:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760760.423314:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512683648:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760760.423416:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6418, 6418] 00000080:00200000:0.0:1789760760.423428:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760760.423433:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760760.423444:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26292224 i_blocks 53255 00000020:00200000:0.0:1789760760.423447:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760760.423455:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760760.423481:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26292224, count: 4096 00000080:00200000:0.0:1789760760.423487:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26292224, bytes: 4096 00000080:00200000:0.0:1789760760.423496:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760760.423503:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760760.423505:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26292224, 26296320) 0 0 00000080:00200000:0.0:1789760760.423508:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760760.423520:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6419, 6419] 00020000:00200000:0.0:1789760760.423526:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26292224 chunk: [26292224, 26296320) 26296320, 4096 00020000:00200000:0.0:1789760760.423530:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760760.423544:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6419, 6419] 00000008:00010000:0.0:1789760760.423556:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760760.423560:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760760.423569:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26292224, 26296320) append 0 append lockpos 0 00000080:00200000:0.0:1789760760.423580:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26292224=0x1913000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760760.423619:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760760.423622:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26296320 00000020:00010000:0.0:1789760760.423628:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760760.423632:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.423635:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.423638:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760760.423641:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760760.423647:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760760.423652:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760760.423730:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccb40 00000008:00100000:0.0:1789760760.423974:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514ad80 - object 0x240000400:5 offset 26292224<>26296320 00000100:00100000:0.0:1789760760.424008:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760760.424022:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512683776:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760760.489521:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973738 gen 3 00000008:00010000:0.0:1789760760.489556:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:0.0:1789760760.489562:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760760.489582:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760760.489600:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:0.0:1789760760.489624:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512683776:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760760.489764:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6419, 6419] 00000080:00200000:3.0:1789760760.489776:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760760.489782:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760760.489793:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26296320 i_blocks 53255 00000020:00200000:3.0:1789760760.489797:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760760.489805:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760760.489829:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26296320, count: 4096 00000080:00200000:3.0:1789760760.489844:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26296320, bytes: 4096 00000080:00200000:3.0:1789760760.489852:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760760.489860:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760760.489862:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26296320, 26300416) 0 0 00000080:00200000:3.0:1789760760.489866:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760760.489880:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6420, 6420] 00020000:00200000:3.0:1789760760.489887:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26296320 chunk: [26296320, 26300416) 26300416, 4096 00020000:00200000:3.0:1789760760.489891:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760760.489930:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6420, 6420] 00000008:00010000:3.0:1789760760.489943:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760760.489947:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760760.489957:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26296320, 26300416) append 0 append lockpos 0 00000080:00200000:3.0:1789760760.489966:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26296320=0x1914000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760760.490006:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760760.490009:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26300416 00000020:00010000:3.0:1789760760.490015:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760760.490017:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760760.490020:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760760.490023:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760760.490025:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760760.490031:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760760.490036:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760760.490049:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760760.490125:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 26296320<>26300416 00000100:00100000:2.0:1789760760.490400:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512683904:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760760.550695:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973739 gen 3 00000008:00010000:3.0:1789760760.550731:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:3.0:1789760760.550737:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760760.550746:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760760.550768:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:3.0:1789760760.550789:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512683904:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760760.551326:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6420, 6420] 00000080:00200000:0.0:1789760760.551340:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760760.551345:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760760.551356:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26300416 i_blocks 53255 00000020:00200000:0.0:1789760760.551359:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760760.551368:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760760.551393:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26300416, count: 4096 00000080:00200000:0.0:1789760760.551399:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26300416, bytes: 4096 00000080:00200000:0.0:1789760760.551408:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760760.551415:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760760.551417:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26300416, 26304512) 0 0 00000080:00200000:0.0:1789760760.551420:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760760.551433:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6421, 6421] 00020000:00200000:0.0:1789760760.551440:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26300416 chunk: [26300416, 26304512) 26304512, 4096 00020000:00200000:0.0:1789760760.551444:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760760.551460:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6421, 6421] 00000008:00010000:0.0:1789760760.551473:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760760.551478:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760760.551486:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26300416, 26304512) append 0 append lockpos 0 00000080:00200000:0.0:1789760760.551497:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26300416=0x1915000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760760.551540:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760760.551542:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26304512 00000020:00010000:0.0:1789760760.551548:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760760.551551:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.551554:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.551557:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760760.551559:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760760.551565:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760760.551571:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760760.551586:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccb40 00000008:00100000:0.0:1789760760.551674:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514ad80 - object 0x240000400:5 offset 26300416<>26304512 00000100:00100000:1.0:1789760760.551748:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512684032:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760760.587954:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760760.587966:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760760.587975:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760760.587992:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760760.588013:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512684032:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760760.588030:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6421, 6421] 00000080:00200000:0.0:1789760760.588042:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760760.588047:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760760.588059:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26304512 i_blocks 53255 00000020:00200000:0.0:1789760760.588063:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760760.588070:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760760.588100:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26304512, count: 4096 00000080:00200000:0.0:1789760760.588107:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26304512, bytes: 4096 00000080:00200000:0.0:1789760760.588114:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760760.588121:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760760.588123:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26304512, 26308608) 0 0 00000080:00200000:0.0:1789760760.588127:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760760.588140:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6422, 6422] 00020000:00200000:0.0:1789760760.588148:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26304512 chunk: [26304512, 26308608) 26308608, 4096 00020000:00200000:0.0:1789760760.588217:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760760.588340:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6422, 6422] 00000008:00010000:0.0:1789760760.588352:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760760.588356:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760760.588364:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26304512, 26308608) append 0 append lockpos 0 00000080:00200000:0.0:1789760760.588374:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26304512=0x1916000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760760.588412:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760760.588414:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26308608 00000020:00010000:0.0:1789760760.588856:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760760.588861:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.588865:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.588868:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760760.588870:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760760.588879:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760760.588884:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760760.590587:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc300 00000008:00100000:0.0:1789760760.591019:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148e00 - object 0x240000400:5 offset 26304512<>26308608 00000100:00100000:1.0:1789760760.591090:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512684160:192.168.203.156@tcp:4:dd.0 00010000:00010000:3.0:1789760760.605088:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760760.626516:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973740 gen 3 00000100:00100000:1.0:1789760760.626546:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005148e00 x1876699512684160/t4294973741(4294973741) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760776 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760760.626585:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760760.626590:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760760.626598:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760760.626622:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760760.626642:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512684160:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760760.626773:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6422, 6422] 00000080:00200000:0.0:1789760760.626785:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760760.626790:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760760.626802:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26308608 i_blocks 53255 00000020:00200000:0.0:1789760760.626805:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760760.626813:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760760.626838:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26308608, count: 4096 00000080:00200000:0.0:1789760760.626843:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26308608, bytes: 4096 00000080:00200000:0.0:1789760760.626850:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760760.626857:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760760.626859:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26308608, 26312704) 0 0 00000080:00200000:0.0:1789760760.626862:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760760.626875:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6423, 6423] 00020000:00200000:0.0:1789760760.626882:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26308608 chunk: [26308608, 26312704) 26312704, 4096 00020000:00200000:0.0:1789760760.626886:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760760.626921:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6423, 6423] 00000008:00010000:0.0:1789760760.626934:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760760.626938:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760760.626947:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26308608, 26312704) append 0 append lockpos 0 00000080:00200000:0.0:1789760760.626965:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26308608=0x1917000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760760.627007:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760760.627009:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26312704 00000020:00010000:0.0:1789760760.627016:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760760.627019:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.627021:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.627024:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760760.627026:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760760.627032:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760760.627037:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760760.627049:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc240 00000008:00100000:0.0:1789760760.627122:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148a80 - object 0x240000400:5 offset 26308608<>26312704 00000100:00100000:0.0:1789760760.627159:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512684288:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760760.706172:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973742 gen 3 00000008:00010000:0.0:1789760760.706218:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:0.0:1789760760.706225:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760760.706233:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760760.706254:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:0.0:1789760760.706291:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512684288:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760760.706534:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6423, 6423] 00000080:00200000:3.0:1789760760.706546:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760760.706551:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760760.706561:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26312704 i_blocks 53255 00000020:00200000:3.0:1789760760.706565:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760760.706572:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760760.706595:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26312704, count: 4096 00000080:00200000:3.0:1789760760.706601:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26312704, bytes: 4096 00000080:00200000:3.0:1789760760.706608:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760760.706616:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760760.706618:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26312704, 26316800) 0 0 00000080:00200000:3.0:1789760760.706621:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760760.706632:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6424, 6424] 00020000:00200000:3.0:1789760760.706638:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26312704 chunk: [26312704, 26316800) 26316800, 4096 00020000:00200000:3.0:1789760760.706642:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760760.706656:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6424, 6424] 00000008:00010000:3.0:1789760760.706728:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760760.706733:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760760.706742:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26312704, 26316800) append 0 append lockpos 0 00000080:00200000:3.0:1789760760.706751:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26312704=0x1918000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760760.706844:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760760.706847:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26316800 00000020:00010000:3.0:1789760760.706853:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760760.706857:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760760.706860:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760760.706863:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760760.706865:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760760.706871:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760760.706876:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760760.706888:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760760.707127:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 26312704<>26316800 00000100:00100000:3.0:1789760760.707162:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512684416:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760760.733297:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973743 gen 3 00000008:00010000:3.0:1789760760.733332:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:3.0:1789760760.733339:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760760.733348:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760760.733409:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:3.0:1789760760.733430:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512684416:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760760.733499:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6424, 6424] 00000080:00200000:2.0:1789760760.733512:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760760.733517:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760760.733528:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26316800 i_blocks 53255 00000020:00200000:2.0:1789760760.733532:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760760.733541:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760760.733566:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26316800, count: 4096 00000080:00200000:2.0:1789760760.733571:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26316800, bytes: 4096 00000080:00200000:2.0:1789760760.733579:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760760.733586:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760760.733588:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26316800, 26320896) 0 0 00000080:00200000:2.0:1789760760.733592:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760760.733604:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6425, 6425] 00020000:00200000:2.0:1789760760.733610:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26316800 chunk: [26316800, 26320896) 26320896, 4096 00020000:00200000:2.0:1789760760.733614:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760760.733628:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6425, 6425] 00000008:00010000:2.0:1789760760.733640:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760760.733645:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760760.733653:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26316800, 26320896) append 0 append lockpos 0 00000080:00200000:2.0:1789760760.733663:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26316800=0x1919000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760760.733782:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760760.733784:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26320896 00000020:00010000:2.0:1789760760.733848:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760760.733852:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.733855:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760760.733858:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760760.733861:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760760.733867:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760760.733873:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760760.733887:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760760.734154:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 26316800<>26320896 00000100:00100000:2.0:1789760760.734185:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760760.734199:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512684544:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760760.771462:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760760.771473:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760760.771483:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760760.771503:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760760.771527:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512684544:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760760.771617:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6425, 6425] 00000080:00200000:0.0:1789760760.771638:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760760.771644:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760760.771656:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26320896 i_blocks 53255 00000020:00200000:0.0:1789760760.771660:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760760.771668:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760760.771703:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26320896, count: 4096 00000080:00200000:0.0:1789760760.771709:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26320896, bytes: 4096 00000080:00200000:0.0:1789760760.771717:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760760.771723:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760760.771725:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26320896, 26324992) 0 0 00000080:00200000:0.0:1789760760.771729:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760760.771741:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6426, 6426] 00020000:00200000:0.0:1789760760.771748:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26320896 chunk: [26320896, 26324992) 26324992, 4096 00020000:00200000:0.0:1789760760.771751:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760760.771767:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6426, 6426] 00000008:00010000:0.0:1789760760.771781:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760760.771786:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760760.771802:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26320896, 26324992) append 0 append lockpos 0 00000080:00200000:0.0:1789760760.771814:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26320896=0x191a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760760.771860:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760760.771863:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26324992 00000020:00010000:0.0:1789760760.771869:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760760.771873:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.771875:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.771879:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760760.771882:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760760.771889:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760760.771895:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760760.771946:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc240 00000008:00100000:0.0:1789760760.772021:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148a80 - object 0x240000400:5 offset 26320896<>26324992 00000100:00100000:0.0:1789760760.772048:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760760.772061:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512684672:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760760.818301:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973745 gen 3 00000008:00010000:0.0:1789760760.818345:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:0.0:1789760760.818352:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760760.818363:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760760.818384:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:0.0:1789760760.818404:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512684672:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760760.818486:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6426, 6426] 00000080:00200000:3.0:1789760760.818499:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760760.818503:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760760.818515:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26324992 i_blocks 53255 00000020:00200000:3.0:1789760760.818518:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760760.818525:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760760.818552:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26324992, count: 4096 00000080:00200000:3.0:1789760760.818558:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26324992, bytes: 4096 00000080:00200000:3.0:1789760760.818565:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760760.818573:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760760.818575:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26324992, 26329088) 0 0 00000080:00200000:3.0:1789760760.818579:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760760.818591:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6427, 6427] 00020000:00200000:3.0:1789760760.818599:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26324992 chunk: [26324992, 26329088) 26329088, 4096 00020000:00200000:3.0:1789760760.818602:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760760.818617:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6427, 6427] 00000008:00010000:3.0:1789760760.818639:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760760.818645:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760760.818654:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26324992, 26329088) append 0 append lockpos 0 00000080:00200000:3.0:1789760760.818663:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26324992=0x191b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760760.818715:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760760.818719:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26329088 00000020:00010000:3.0:1789760760.818726:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760760.818729:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760760.818732:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760760.818735:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760760.818738:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760760.818744:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760760.818749:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760760.818763:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760760.818834:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 26324992<>26329088 00000100:00100000:2.0:1789760760.819032:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512684800:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760760.854021:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973746 gen 3 00000008:00010000:3.0:1789760760.854061:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:3.0:1789760760.854068:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760760.854077:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760760.854101:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:3.0:1789760760.854124:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512684800:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760760.854333:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6427, 6427] 00000080:00200000:0.0:1789760760.854346:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760760.854351:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760760.854363:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26329088 i_blocks 53255 00000020:00200000:0.0:1789760760.854367:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760760.854374:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760760.854399:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26329088, count: 4096 00000080:00200000:0.0:1789760760.854405:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26329088, bytes: 4096 00000080:00200000:0.0:1789760760.854412:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760760.854419:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760760.854421:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26329088, 26333184) 0 0 00000080:00200000:0.0:1789760760.854425:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760760.854438:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6428, 6428] 00020000:00200000:0.0:1789760760.854444:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26329088 chunk: [26329088, 26333184) 26333184, 4096 00020000:00200000:0.0:1789760760.854449:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760760.854465:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6428, 6428] 00000008:00010000:0.0:1789760760.854480:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760760.854485:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760760.854494:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26329088, 26333184) append 0 append lockpos 0 00000080:00200000:0.0:1789760760.854503:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26329088=0x191c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760760.854545:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760760.854548:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26333184 00000020:00010000:0.0:1789760760.854555:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760760.854558:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.854561:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.854565:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760760.854567:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760760.854573:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760760.854578:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760760.854591:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc240 00000008:00100000:0.0:1789760760.854662:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148a80 - object 0x240000400:5 offset 26329088<>26333184 00000100:00100000:1.0:1789760760.855133:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512684928:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760760.899562:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760760.899575:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760760.899585:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760760.899606:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760760.899629:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512684928:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760760.899767:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6428, 6428] 00000080:00200000:0.0:1789760760.899872:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760760.899878:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760760.899889:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26333184 i_blocks 53255 00000020:00200000:0.0:1789760760.899892:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760760.899925:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760760.900009:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26333184, count: 4096 00000080:00200000:0.0:1789760760.900016:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26333184, bytes: 4096 00000080:00200000:0.0:1789760760.900024:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760760.900030:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760760.900032:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26333184, 26337280) 0 0 00000080:00200000:0.0:1789760760.900037:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760760.900049:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6429, 6429] 00020000:00200000:0.0:1789760760.900056:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26333184 chunk: [26333184, 26337280) 26337280, 4096 00020000:00200000:0.0:1789760760.900060:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760760.900076:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6429, 6429] 00000008:00010000:0.0:1789760760.900090:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760760.900095:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760760.900104:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26333184, 26337280) append 0 append lockpos 0 00000080:00200000:0.0:1789760760.900116:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26333184=0x191d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760760.900159:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760760.900162:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26337280 00000020:00010000:0.0:1789760760.900168:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760760.900171:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.900175:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.900178:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760760.900181:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760760.900187:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760760.900193:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760760.900207:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc480 00000008:00100000:0.0:1789760760.900296:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148e00 - object 0x240000400:5 offset 26333184<>26337280 00000100:00100000:1.0:1789760760.900463:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512685056:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760760.944163:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973748 gen 3 00000008:00010000:1.0:1789760760.944211:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760760.944217:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760760.944225:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760760.944247:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760760.944269:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512685056:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760760.944296:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6429, 6429] 00000080:00200000:0.0:1789760760.944309:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760760.944315:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760760.944327:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26337280 i_blocks 53255 00000020:00200000:0.0:1789760760.944331:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760760.944339:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760760.944362:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26337280, count: 4096 00000080:00200000:0.0:1789760760.944368:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26337280, bytes: 4096 00000080:00200000:0.0:1789760760.944375:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760760.944382:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760760.944385:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26337280, 26341376) 0 0 00000080:00200000:0.0:1789760760.944388:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760760.944401:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6430, 6430] 00020000:00200000:0.0:1789760760.944408:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26337280 chunk: [26337280, 26341376) 26341376, 4096 00020000:00200000:0.0:1789760760.944412:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760760.944426:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6430, 6430] 00000008:00010000:0.0:1789760760.944439:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760760.944443:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760760.944452:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26337280, 26341376) append 0 append lockpos 0 00000080:00200000:0.0:1789760760.944462:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26337280=0x191e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760760.944500:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760760.944502:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26341376 00000020:00010000:0.0:1789760760.944508:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760760.944510:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.944512:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760760.944514:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760760.944516:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760760.944520:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760760.944524:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760760.944535:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc840 00000008:00100000:0.0:1789760760.944594:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a680 - object 0x240000400:5 offset 26337280<>26341376 00000100:00100000:0.0:1789760760.944626:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760760.944638:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512685184:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760761.031812:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973749 gen 3 00000008:00010000:0.0:1789760761.031850:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:0.0:1789760761.031857:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760761.031866:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760761.031887:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:0.0:1789760761.031960:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512685184:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760761.032885:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6430, 6430] 00000080:00200000:3.0:1789760761.032897:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.032920:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.032931:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26341376 i_blocks 53255 00000020:00200000:3.0:1789760761.032934:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.032943:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.032965:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26341376, count: 4096 00000080:00200000:3.0:1789760761.032971:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26341376, bytes: 4096 00000080:00200000:3.0:1789760761.032990:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.032997:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.032999:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26341376, 26345472) 0 0 00000080:00200000:3.0:1789760761.033003:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.033015:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6431, 6431] 00020000:00200000:3.0:1789760761.033021:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26341376 chunk: [26341376, 26345472) 26345472, 4096 00020000:00200000:3.0:1789760761.033025:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.033038:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6431, 6431] 00000008:00010000:3.0:1789760761.033055:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.033059:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760761.033068:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26341376, 26345472) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.033080:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26341376=0x191f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.033119:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.033122:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26345472 00000020:00010000:3.0:1789760761.033128:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.033131:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.033134:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.033137:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.033139:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.033145:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.033149:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.033162:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760761.033232:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 26341376<>26345472 00000100:00100000:2.0:1789760761.033309:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512685312:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760761.085457:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973750 gen 3 00000008:00010000:3.0:1789760761.085502:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:3.0:1789760761.085509:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760761.085519:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760761.085543:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:3.0:1789760761.085567:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512685312:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760761.085710:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6431, 6431] 00000080:00200000:0.0:1789760761.085722:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760761.085727:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760761.085739:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26345472 i_blocks 53255 00000020:00200000:0.0:1789760761.085742:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760761.085750:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760761.085774:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26345472, count: 4096 00000080:00200000:0.0:1789760761.085780:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26345472, bytes: 4096 00000080:00200000:0.0:1789760761.085788:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760761.085795:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760761.085797:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26345472, 26349568) 0 0 00000080:00200000:0.0:1789760761.085801:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760761.085812:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6432, 6432] 00020000:00200000:0.0:1789760761.085819:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26345472 chunk: [26345472, 26349568) 26349568, 4096 00020000:00200000:0.0:1789760761.085823:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760761.085836:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6432, 6432] 00000008:00010000:0.0:1789760761.085850:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760761.085854:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760761.085863:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26345472, 26349568) append 0 append lockpos 0 00000080:00200000:0.0:1789760761.085874:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26345472=0x1920000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760761.085973:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760761.085975:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26349568 00000020:00010000:0.0:1789760761.085982:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760761.085985:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.085988:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.085991:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760761.085994:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760761.086000:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760761.086005:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760761.086015:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc840 00000008:00100000:0.0:1789760761.086084:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a680 - object 0x240000400:5 offset 26345472<>26349568 00000100:00100000:1.0:1789760761.086239:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512685440:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760761.138014:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973751 gen 3 00000008:00010000:1.0:1789760761.138053:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760761.138059:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760761.138068:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760761.138090:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760761.138111:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512685440:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760761.138314:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6432, 6432] 00000080:00200000:0.0:1789760761.138327:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760761.138332:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760761.138343:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26349568 i_blocks 53255 00000020:00200000:0.0:1789760761.138347:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760761.138355:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760761.138382:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26349568, count: 4096 00000080:00200000:0.0:1789760761.138388:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26349568, bytes: 4096 00000080:00200000:0.0:1789760761.138395:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760761.138402:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760761.138404:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26349568, 26353664) 0 0 00000080:00200000:0.0:1789760761.138408:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760761.138420:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6433, 6433] 00020000:00200000:0.0:1789760761.138427:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26349568 chunk: [26349568, 26353664) 26353664, 4096 00020000:00200000:0.0:1789760761.138431:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760761.138446:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6433, 6433] 00000008:00010000:0.0:1789760761.138461:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760761.138465:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760761.138474:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26349568, 26353664) append 0 append lockpos 0 00000080:00200000:0.0:1789760761.138487:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26349568=0x1921000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760761.138530:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760761.138532:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26353664 00000020:00010000:0.0:1789760761.138540:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760761.138543:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.138545:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.138548:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760761.138551:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760761.138557:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760761.138562:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760761.138576:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc840 00000008:00100000:0.0:1789760761.138661:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a300 - object 0x240000400:5 offset 26349568<>26353664 00000100:00100000:1.0:1789760761.138827:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512685568:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760761.167877:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973752 gen 3 00000008:00010000:1.0:1789760761.167939:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760761.167945:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760761.167954:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760761.167974:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760761.167996:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512685568:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760761.168089:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6433, 6433] 00000080:00200000:0.0:1789760761.168100:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760761.168105:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760761.168117:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26353664 i_blocks 53255 00000020:00200000:0.0:1789760761.168120:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760761.168127:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760761.168152:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26353664, count: 4096 00000080:00200000:0.0:1789760761.168158:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26353664, bytes: 4096 00000080:00200000:0.0:1789760761.168165:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760761.168173:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760761.168175:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26353664, 26357760) 0 0 00000080:00200000:0.0:1789760761.168178:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760761.168192:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6434, 6434] 00020000:00200000:0.0:1789760761.168199:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26353664 chunk: [26353664, 26357760) 26357760, 4096 00020000:00200000:0.0:1789760761.168203:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760761.168217:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6434, 6434] 00000008:00010000:0.0:1789760761.168230:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760761.168234:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760761.168243:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26353664, 26357760) append 0 append lockpos 0 00000080:00200000:0.0:1789760761.168254:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26353664=0x1922000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760761.168306:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760761.168309:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26357760 00000020:00010000:0.0:1789760761.168315:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760761.168318:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.168321:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.168324:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760761.168326:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760761.168332:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760761.168338:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760761.168352:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc480 00000008:00100000:0.0:1789760761.168426:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149880 - object 0x240000400:5 offset 26353664<>26357760 00000100:00100000:1.0:1789760761.168524:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512685696:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760761.206123:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973753 gen 3 00000008:00010000:1.0:1789760761.206162:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760761.206168:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760761.206176:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760761.206246:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760761.206270:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512685696:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760761.206404:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6434, 6434] 00000080:00200000:0.0:1789760761.206416:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760761.206421:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760761.206432:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26357760 i_blocks 53255 00000020:00200000:0.0:1789760761.206435:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760761.206442:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760761.206468:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26357760, count: 4096 00000080:00200000:0.0:1789760761.206473:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26357760, bytes: 4096 00000080:00200000:0.0:1789760761.206480:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760761.206487:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760761.206489:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26357760, 26361856) 0 0 00000080:00200000:0.0:1789760761.206492:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760761.206561:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6435, 6435] 00020000:00200000:0.0:1789760761.206568:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26357760 chunk: [26357760, 26361856) 26361856, 4096 00020000:00200000:0.0:1789760761.206571:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760761.206587:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6435, 6435] 00000008:00010000:0.0:1789760761.206600:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760761.206604:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760761.206613:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26357760, 26361856) append 0 append lockpos 0 00000080:00200000:0.0:1789760761.206622:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26357760=0x1923000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760761.206766:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760761.206769:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26361856 00000020:00010000:0.0:1789760761.206777:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760761.206781:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.206784:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.206787:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760761.206789:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760761.206796:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760761.206801:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760761.206815:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc240 00000008:00100000:0.0:1789760761.207142:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b480 - object 0x240000400:5 offset 26357760<>26361856 00000100:00100000:1.0:1789760761.207196:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512685824:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760761.257353:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973754 gen 3 00000008:00010000:1.0:1789760761.257391:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760761.257398:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760761.257407:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760761.257429:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760761.257451:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512685824:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760761.257509:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6435, 6435] 00000080:00200000:0.0:1789760761.257522:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760761.257527:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760761.257538:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26361856 i_blocks 53255 00000020:00200000:0.0:1789760761.257541:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760761.257549:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760761.257574:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26361856, count: 4096 00000080:00200000:0.0:1789760761.257580:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26361856, bytes: 4096 00000080:00200000:0.0:1789760761.257588:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760761.257597:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760761.257599:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26361856, 26365952) 0 0 00000080:00200000:0.0:1789760761.257602:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760761.257616:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6436, 6436] 00020000:00200000:0.0:1789760761.257623:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26361856 chunk: [26361856, 26365952) 26365952, 4096 00020000:00200000:0.0:1789760761.257627:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760761.257644:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6436, 6436] 00000008:00010000:0.0:1789760761.257659:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760761.257664:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760761.257745:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26361856, 26365952) append 0 append lockpos 0 00000080:00200000:0.0:1789760761.257757:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26361856=0x1924000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760761.257800:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760761.257803:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26365952 00000020:00010000:0.0:1789760761.257810:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760761.257813:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.257816:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.257818:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760761.257821:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760761.257827:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760761.257833:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760761.257846:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc300 00000008:00100000:0.0:1789760761.257957:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148380 - object 0x240000400:5 offset 26361856<>26365952 00000100:00100000:1.0:1789760761.258003:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512685952:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760761.330405:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973755 gen 3 00000008:00010000:1.0:1789760761.330444:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760761.330451:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760761.330459:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760761.330480:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760761.330506:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512685952:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760761.330748:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6436, 6436] 00000080:00200000:0.0:1789760761.330762:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760761.330767:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760761.330780:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26365952 i_blocks 53255 00000020:00200000:0.0:1789760761.330783:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760761.330792:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760761.330820:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26365952, count: 4096 00000080:00200000:0.0:1789760761.330826:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26365952, bytes: 4096 00000080:00200000:0.0:1789760761.330834:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760761.330892:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760761.330894:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26365952, 26370048) 0 0 00000080:00200000:0.0:1789760761.330898:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760761.330942:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6437, 6437] 00020000:00200000:0.0:1789760761.330950:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26365952 chunk: [26365952, 26370048) 26370048, 4096 00020000:00200000:0.0:1789760761.330953:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760761.330969:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6437, 6437] 00000008:00010000:0.0:1789760761.330983:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760761.330988:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760761.330997:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26365952, 26370048) append 0 append lockpos 0 00000080:00200000:0.0:1789760761.331007:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26365952=0x1925000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760761.331053:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760761.331056:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26370048 00000020:00010000:0.0:1789760761.331062:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760761.331065:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.331067:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.331070:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760761.331072:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760761.331078:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760761.331083:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760761.331097:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccb40 00000008:00100000:0.0:1789760761.331169:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b800 - object 0x240000400:5 offset 26365952<>26370048 00000100:00100000:1.0:1789760761.331245:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512686080:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760761.371225:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973756 gen 3 00000008:00010000:1.0:1789760761.371259:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760761.371266:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760761.371402:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760761.371483:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000080:00200000:0.0:1789760761.371697:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6437, 6437] 00000080:00200000:0.0:1789760761.371710:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760761.371715:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760761.371732:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26370048 i_blocks 53255 00000020:00200000:0.0:1789760761.371736:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760761.371744:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760761.371770:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26370048, count: 4096 00000080:00200000:0.0:1789760761.371777:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26370048, bytes: 4096 00000080:00200000:0.0:1789760761.371836:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760761.371843:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760761.371845:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26370048, 26374144) 0 0 00000080:00200000:0.0:1789760761.371849:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760761.371862:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6438, 6438] 00020000:00200000:0.0:1789760761.371868:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26370048 chunk: [26370048, 26374144) 26374144, 4096 00020000:00200000:0.0:1789760761.371872:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760761.371887:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6438, 6438] 00000008:00010000:0.0:1789760761.371930:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760761.371936:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760761.371946:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26370048, 26374144) append 0 append lockpos 0 00000080:00200000:0.0:1789760761.371959:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26370048=0x1926000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760761.372014:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760761.372016:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26374144 00000020:00010000:0.0:1789760761.372023:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760761.372026:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.372029:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.372032:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760761.372034:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760761.372040:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760761.372045:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760761.372064:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760761.372073:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512686080:192.168.203.156@tcp:4:dd.0 00000008:00100000:0.0:1789760761.373434:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149c00 - object 0x240000400:5 offset 26370048<>26374144 00000100:00100000:0.0:1789760761.373515:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512686208:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760761.404047:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973757 gen 3 00000008:00010000:0.0:1789760761.404081:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:0.0:1789760761.404087:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760761.404095:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760761.404113:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:0.0:1789760761.404134:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512686208:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760761.404982:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6438, 6438] 00000080:00200000:1.0:1789760761.404995:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760761.405000:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760761.405012:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26374144 i_blocks 53255 00000020:00200000:1.0:1789760761.405015:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760761.405025:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760761.405049:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26374144, count: 4096 00000080:00200000:1.0:1789760761.405055:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26374144, bytes: 4096 00000080:00200000:1.0:1789760761.405064:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760761.405071:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760761.405073:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26374144, 26378240) 0 0 00000080:00200000:1.0:1789760761.405076:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760761.405090:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6439, 6439] 00020000:00200000:1.0:1789760761.405096:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26374144 chunk: [26374144, 26378240) 26378240, 4096 00020000:00200000:1.0:1789760761.405100:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760761.405114:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6439, 6439] 00000008:00010000:1.0:1789760761.405128:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760761.405133:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760761.405142:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26374144, 26378240) append 0 append lockpos 0 00000080:00200000:1.0:1789760761.405163:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26374144=0x1927000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760761.405211:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760761.405214:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26378240 00000020:00010000:1.0:1789760761.405220:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760761.405224:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760761.405226:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760761.405229:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760761.405232:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760761.405237:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760761.405244:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760761.405267:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760761.405353:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97100 - object 0x240000400:5 offset 26374144<>26378240 00000100:00100000:0.0:1789760761.405675:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512686336:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760761.432137:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760761.432149:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760761.432157:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760761.432177:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760761.432199:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512686336:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760761.432298:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6439, 6439] 00000080:00200000:1.0:1789760761.432310:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760761.432315:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760761.432335:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26378240 i_blocks 53255 00000020:00200000:1.0:1789760761.432339:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760761.432346:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760761.432372:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26378240, count: 4096 00000080:00200000:1.0:1789760761.432377:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26378240, bytes: 4096 00000080:00200000:1.0:1789760761.432384:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760761.432391:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760761.432393:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26378240, 26382336) 0 0 00000080:00200000:1.0:1789760761.432396:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760761.432408:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6440, 6440] 00020000:00200000:1.0:1789760761.432414:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26378240 chunk: [26378240, 26382336) 26382336, 4096 00020000:00200000:1.0:1789760761.432418:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760761.432433:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6440, 6440] 00000008:00010000:1.0:1789760761.432446:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760761.432451:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760761.432460:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26378240, 26382336) append 0 append lockpos 0 00000080:00200000:1.0:1789760761.432470:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26378240=0x1928000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760761.432511:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760761.432514:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26382336 00000020:00010000:1.0:1789760761.432520:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760761.432524:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760761.432527:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760761.432530:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760761.432532:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760761.432537:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760761.432543:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760761.432557:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760761.432642:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 26378240<>26382336 00000100:00100000:0.0:1789760761.432846:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512686464:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760761.475160:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973758 gen 3 00000100:00100000:0.0:1789760761.475194:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005c94a80 x1876699512686464/t4294973759(4294973759) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760777 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760761.475233:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760761.475239:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760761.475247:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760761.475269:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760761.475298:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512686464:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760761.475439:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6440, 6440] 00000080:00200000:3.0:1789760761.475452:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.475457:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.475468:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26382336 i_blocks 53255 00000020:00200000:3.0:1789760761.475471:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.475478:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.475504:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26382336, count: 4096 00000080:00200000:3.0:1789760761.475511:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26382336, bytes: 4096 00000080:00200000:3.0:1789760761.475519:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.475526:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.475528:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26382336, 26386432) 0 0 00000080:00200000:3.0:1789760761.475532:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.475546:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6441, 6441] 00020000:00200000:3.0:1789760761.475553:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26382336 chunk: [26382336, 26386432) 26386432, 4096 00020000:00200000:3.0:1789760761.475556:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.475571:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6441, 6441] 00000008:00010000:3.0:1789760761.475584:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.475589:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760761.475607:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26382336, 26386432) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.475619:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26382336=0x1929000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.475663:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.475666:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26386432 00000020:00010000:3.0:1789760761.475673:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.475676:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.475678:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.475681:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.475683:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.475689:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.475693:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.475706:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760761.475797:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 26382336<>26386432 00000100:00100000:3.0:1789760761.475835:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760761.475848:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512686592:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760761.508158:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973760 gen 3 00000008:00010000:3.0:1789760761.508203:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:3.0:1789760761.508210:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760761.508219:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760761.508244:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:3.0:1789760761.508268:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512686592:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760761.508331:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6441, 6441] 00000080:00200000:0.0:1789760761.508344:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760761.508349:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760761.508360:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26386432 i_blocks 53255 00000020:00200000:0.0:1789760761.508364:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760761.508372:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760761.508398:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26386432, count: 4096 00000080:00200000:0.0:1789760761.508404:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26386432, bytes: 4096 00000080:00200000:0.0:1789760761.508412:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760761.508419:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760761.508421:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26386432, 26390528) 0 0 00000080:00200000:0.0:1789760761.508425:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760761.508439:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6442, 6442] 00020000:00200000:0.0:1789760761.508445:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26386432 chunk: [26386432, 26390528) 26390528, 4096 00020000:00200000:0.0:1789760761.508449:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760761.508464:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6442, 6442] 00000008:00010000:0.0:1789760761.508477:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760761.508481:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760761.508490:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26386432, 26390528) append 0 append lockpos 0 00000080:00200000:0.0:1789760761.508501:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26386432=0x192a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760761.508543:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760761.508546:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26390528 00000020:00010000:0.0:1789760761.508552:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760761.508554:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.508557:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.508560:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760761.508562:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760761.508568:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760761.508573:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760761.508585:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000008:00100000:0.0:1789760761.508658:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149c00 - object 0x240000400:5 offset 26386432<>26390528 00000100:00100000:0.0:1789760761.508698:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760761.508712:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512686720:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760761.544961:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973761 gen 3 00000008:00010000:0.0:1789760761.544998:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:0.0:1789760761.545005:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760761.545014:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760761.545033:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:0.0:1789760761.545056:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512686720:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760761.545187:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6442, 6442] 00000080:00200000:3.0:1789760761.545199:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.545204:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.545215:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26390528 i_blocks 53255 00000020:00200000:3.0:1789760761.545218:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.545226:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.545252:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26390528, count: 4096 00000080:00200000:3.0:1789760761.545257:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26390528, bytes: 4096 00000080:00200000:3.0:1789760761.545264:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.545271:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.545282:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26390528, 26394624) 0 0 00000080:00200000:3.0:1789760761.545286:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.545299:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6443, 6443] 00020000:00200000:3.0:1789760761.545306:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26390528 chunk: [26390528, 26394624) 26394624, 4096 00020000:00200000:3.0:1789760761.545309:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.545323:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6443, 6443] 00000008:00010000:3.0:1789760761.545337:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.545341:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760761.545350:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26390528, 26394624) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.545360:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26390528=0x192b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.545402:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.545405:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26394624 00000020:00010000:3.0:1789760761.545411:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.545414:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.545417:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.545420:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.545423:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.545429:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.545435:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.545447:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760761.545518:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 26390528<>26394624 00000100:00100000:2.0:1789760761.546265:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760761.546291:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512686848:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760761.577651:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973762 gen 3 00000008:00010000:3.0:1789760761.577698:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:3.0:1789760761.577705:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760761.577714:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760761.577735:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:3.0:1789760761.577758:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512686848:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760761.578094:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6443, 6443] 00000080:00200000:0.0:1789760761.578107:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760761.578112:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760761.578123:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26394624 i_blocks 53255 00000020:00200000:0.0:1789760761.578127:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760761.578134:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760761.578159:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26394624, count: 4096 00000080:00200000:0.0:1789760761.578166:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26394624, bytes: 4096 00000080:00200000:0.0:1789760761.578174:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760761.578182:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760761.578184:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26394624, 26398720) 0 0 00000080:00200000:0.0:1789760761.578188:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760761.578201:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6444, 6444] 00020000:00200000:0.0:1789760761.578208:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26394624 chunk: [26394624, 26398720) 26398720, 4096 00020000:00200000:0.0:1789760761.578212:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760761.578226:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6444, 6444] 00000008:00010000:0.0:1789760761.578239:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760761.578244:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760761.578252:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26394624, 26398720) append 0 append lockpos 0 00000080:00200000:0.0:1789760761.578263:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26394624=0x192c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760761.578315:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760761.578318:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26398720 00000020:00010000:0.0:1789760761.578325:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760761.578328:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.578331:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.578335:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760761.578338:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760761.578345:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760761.578350:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760761.578363:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000008:00100000:0.0:1789760761.578436:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149c00 - object 0x240000400:5 offset 26394624<>26398720 00000100:00100000:1.0:1789760761.578572:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760761.578588:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512686976:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760761.607085:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973763 gen 3 00000008:00010000:1.0:1789760761.607143:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760761.607152:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760761.607162:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760761.607181:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760761.607205:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512686976:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760761.607325:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6444, 6444] 00000080:00200000:0.0:1789760761.607343:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760761.607347:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760761.607358:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26398720 i_blocks 53255 00000020:00200000:0.0:1789760761.607361:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760761.607369:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760761.607395:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26398720, count: 4096 00000080:00200000:0.0:1789760761.607401:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26398720, bytes: 4096 00000080:00200000:0.0:1789760761.607408:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760761.607416:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760761.607418:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26398720, 26402816) 0 0 00000080:00200000:0.0:1789760761.607422:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760761.607434:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6445, 6445] 00020000:00200000:0.0:1789760761.607441:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26398720 chunk: [26398720, 26402816) 26402816, 4096 00020000:00200000:0.0:1789760761.607444:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760761.607458:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6445, 6445] 00000008:00010000:0.0:1789760761.607471:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760761.607476:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760761.607485:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26398720, 26402816) append 0 append lockpos 0 00000080:00200000:0.0:1789760761.607496:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26398720=0x192d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760761.607539:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760761.607542:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26402816 00000020:00010000:0.0:1789760761.607549:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760761.607552:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.607555:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.607558:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760761.607561:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760761.607568:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760761.607573:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760761.607586:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc900 00000008:00100000:0.0:1789760761.607658:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149180 - object 0x240000400:5 offset 26398720<>26402816 00000100:00100000:1.0:1789760761.608234:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760761.608251:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512687104:192.168.203.156@tcp:4:dd.0 00010000:00010000:1.0:1789760761.630930:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760761.641996:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973764 gen 3 00000008:00010000:1.0:1789760761.642042:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760761.642050:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760761.642059:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760761.642087:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760761.642111:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512687104:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760761.642212:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6445, 6445] 00000080:00200000:0.0:1789760761.642224:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760761.642230:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760761.642245:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26402816 i_blocks 53255 00000020:00200000:0.0:1789760761.642249:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760761.642257:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760761.642292:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26402816, count: 4096 00000080:00200000:0.0:1789760761.642299:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26402816, bytes: 4096 00000080:00200000:0.0:1789760761.642307:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760761.642314:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760761.642316:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26402816, 26406912) 0 0 00000080:00200000:0.0:1789760761.642320:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760761.642334:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6446, 6446] 00020000:00200000:0.0:1789760761.642340:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26402816 chunk: [26402816, 26406912) 26406912, 4096 00020000:00200000:0.0:1789760761.642344:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760761.642359:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6446, 6446] 00000008:00010000:0.0:1789760761.642373:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760761.642378:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760761.642387:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26402816, 26406912) append 0 append lockpos 0 00000080:00200000:0.0:1789760761.642398:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26402816=0x192e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760761.642441:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760761.642445:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26406912 00000020:00010000:0.0:1789760761.642451:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760761.642454:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.642457:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.642461:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760761.642464:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760761.642470:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760761.642476:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760761.642491:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cccc0 00000008:00100000:0.0:1789760761.642570:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149f80 - object 0x240000400:5 offset 26402816<>26406912 00000100:00100000:1.0:1789760761.643281:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760761.643300:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512687232:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760761.691654:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973765 gen 3 00000008:00010000:1.0:1789760761.691701:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760761.691707:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760761.691716:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760761.691739:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760761.691763:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512687232:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760761.691930:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6446, 6446] 00000080:00200000:0.0:1789760761.691941:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760761.691956:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760761.691969:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26406912 i_blocks 53255 00000020:00200000:0.0:1789760761.691972:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760761.691981:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760761.692007:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26406912, count: 4096 00000080:00200000:0.0:1789760761.692013:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26406912, bytes: 4096 00000080:00200000:0.0:1789760761.692020:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760761.692027:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760761.692029:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26406912, 26411008) 0 0 00000080:00200000:0.0:1789760761.692032:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760761.692045:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6447, 6447] 00020000:00200000:0.0:1789760761.692052:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26406912 chunk: [26406912, 26411008) 26411008, 4096 00020000:00200000:0.0:1789760761.692056:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760761.692071:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6447, 6447] 00000008:00010000:0.0:1789760761.692085:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760761.692089:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760761.692098:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26406912, 26411008) append 0 append lockpos 0 00000080:00200000:0.0:1789760761.692110:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26406912=0x192f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760761.692156:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760761.692158:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26411008 00000020:00010000:0.0:1789760761.692166:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760761.692169:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.692172:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.692175:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760761.692178:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760761.692184:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760761.692190:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760761.692204:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000008:00100000:0.0:1789760761.692293:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148000 - object 0x240000400:5 offset 26406912<>26411008 00000100:00100000:1.0:1789760761.692512:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760761.692526:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512687360:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760761.726471:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973766 gen 3 00000008:00010000:1.0:1789760761.726505:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760761.726512:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760761.726520:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760761.726542:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760761.726564:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512687360:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760761.726843:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6447, 6447] 00000080:00200000:0.0:1789760761.726857:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760761.726862:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760761.726874:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26411008 i_blocks 53255 00000020:00200000:0.0:1789760761.726878:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760761.726886:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760761.726942:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26411008, count: 4096 00000080:00200000:0.0:1789760761.726949:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26411008, bytes: 4096 00000080:00200000:0.0:1789760761.726956:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760761.726963:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760761.726966:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26411008, 26415104) 0 0 00000080:00200000:0.0:1789760761.726970:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760761.726984:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6448, 6448] 00020000:00200000:0.0:1789760761.726991:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26411008 chunk: [26411008, 26415104) 26415104, 4096 00020000:00200000:0.0:1789760761.726995:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760761.727013:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6448, 6448] 00000008:00010000:0.0:1789760761.727028:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760761.727033:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760761.727042:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26411008, 26415104) append 0 append lockpos 0 00000080:00200000:0.0:1789760761.727053:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26411008=0x1930000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760761.727101:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760761.727104:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26415104 00000020:00010000:0.0:1789760761.727111:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760761.727114:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.727117:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.727121:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760761.727123:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760761.727130:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760761.727135:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760761.727150:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc600 00000008:00100000:0.0:1789760761.727375:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149500 - object 0x240000400:5 offset 26411008<>26415104 00000100:00100000:1.0:1789760761.727896:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760761.727945:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512687488:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760761.756140:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760761.756288:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760761.756299:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760761.756360:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6448, 6448] 00000008:00010000:1.0:1789760761.756360:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000080:00200000:0.0:1789760761.756372:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760761.756377:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:1.0:1789760761.756384:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512687488:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760761.756387:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26415104 i_blocks 53255 00000020:00200000:0.0:1789760761.756390:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760761.756398:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760761.756424:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26415104, count: 4096 00000080:00200000:0.0:1789760761.756430:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26415104, bytes: 4096 00000080:00200000:0.0:1789760761.756437:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760761.756445:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760761.756447:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26415104, 26419200) 0 0 00000080:00200000:0.0:1789760761.756450:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760761.756463:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6449, 6449] 00020000:00200000:0.0:1789760761.756470:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26415104 chunk: [26415104, 26419200) 26419200, 4096 00020000:00200000:0.0:1789760761.756474:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760761.756489:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6449, 6449] 00000008:00010000:0.0:1789760761.756501:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760761.756506:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760761.756514:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26415104, 26419200) append 0 append lockpos 0 00000080:00200000:0.0:1789760761.756525:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26415104=0x1931000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760761.756570:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760761.756573:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26419200 00000020:00010000:0.0:1789760761.756579:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760761.756582:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.756585:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.756588:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760761.756591:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760761.756597:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760761.756603:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760761.756615:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc540 00000008:00100000:0.0:1789760761.756690:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514bb80 - object 0x240000400:5 offset 26415104<>26419200 00000100:00100000:1.0:1789760761.756753:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760761.756767:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512687616:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760761.798506:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973768 gen 3 00000008:00010000:1.0:1789760761.798553:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760761.798559:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760761.798568:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760761.798589:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760761.798612:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512687616:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760761.798758:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6449, 6449] 00000080:00200000:0.0:1789760761.798770:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760761.798775:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760761.798788:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26419200 i_blocks 53255 00000020:00200000:0.0:1789760761.798791:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760761.798799:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760761.798821:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26419200, count: 4096 00000080:00200000:0.0:1789760761.798828:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26419200, bytes: 4096 00000080:00200000:0.0:1789760761.798834:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760761.798842:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760761.798844:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26419200, 26423296) 0 0 00000080:00200000:0.0:1789760761.798850:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760761.798862:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6450, 6450] 00020000:00200000:0.0:1789760761.798868:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26419200 chunk: [26419200, 26423296) 26423296, 4096 00020000:00200000:0.0:1789760761.798871:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760761.798885:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6450, 6450] 00000008:00010000:0.0:1789760761.798924:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760761.798930:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760761.798938:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26419200, 26423296) append 0 append lockpos 0 00000080:00200000:0.0:1789760761.798959:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26419200=0x1932000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760761.799000:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760761.799003:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26423296 00000020:00010000:0.0:1789760761.799010:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760761.799013:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.799016:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.799019:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760761.799022:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760761.799028:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760761.799034:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760761.799047:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cca80 00000008:00100000:0.0:1789760761.799176:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b100 - object 0x240000400:5 offset 26419200<>26423296 00000100:00100000:1.0:1789760761.799231:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512687744:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760761.832135:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973769 gen 3 00000008:00010000:1.0:1789760761.832175:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760761.832181:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760761.832189:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760761.832209:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760761.832231:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512687744:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760761.832589:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6450, 6450] 00000080:00200000:0.0:1789760761.832601:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760761.832606:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760761.832618:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26423296 i_blocks 53255 00000020:00200000:0.0:1789760761.832621:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760761.832629:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760761.832654:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26423296, count: 4096 00000080:00200000:0.0:1789760761.832660:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26423296, bytes: 4096 00000080:00200000:0.0:1789760761.832676:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760761.832684:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760761.832686:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26423296, 26427392) 0 0 00000080:00200000:0.0:1789760761.832689:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760761.832703:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6451, 6451] 00020000:00200000:0.0:1789760761.832709:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26423296 chunk: [26423296, 26427392) 26427392, 4096 00020000:00200000:0.0:1789760761.832712:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760761.832727:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6451, 6451] 00000008:00010000:0.0:1789760761.832740:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760761.832745:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760761.832754:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26423296, 26427392) append 0 append lockpos 0 00000080:00200000:0.0:1789760761.832764:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26423296=0x1933000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760761.832818:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760761.832821:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26427392 00000020:00010000:0.0:1789760761.832828:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760761.832831:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.832833:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.832837:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760761.832840:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760761.832846:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760761.832862:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760761.832876:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccf00 00000008:00100000:0.0:1789760761.833187:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b100 - object 0x240000400:5 offset 26423296<>26427392 00000100:00100000:0.0:1789760761.833232:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512687872:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760761.874247:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973770 gen 3 00000008:00010000:0.0:1789760761.874359:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:0.0:1789760761.874366:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760761.874376:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760761.874446:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:0.0:1789760761.874471:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512687872:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760761.874732:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6451, 6451] 00000080:00200000:1.0:1789760761.874746:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760761.874751:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760761.874761:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26427392 i_blocks 53255 00000020:00200000:1.0:1789760761.874765:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760761.874773:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760761.874859:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26427392, count: 4096 00000080:00200000:1.0:1789760761.874866:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26427392, bytes: 4096 00000080:00200000:1.0:1789760761.874875:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760761.874882:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760761.874884:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26427392, 26431488) 0 0 00000080:00200000:1.0:1789760761.874888:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760761.874927:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6452, 6452] 00020000:00200000:1.0:1789760761.874934:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26427392 chunk: [26427392, 26431488) 26431488, 4096 00020000:00200000:1.0:1789760761.874995:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760761.875014:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6452, 6452] 00000008:00010000:1.0:1789760761.875028:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760761.875033:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760761.875042:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26427392, 26431488) append 0 append lockpos 0 00000080:00200000:1.0:1789760761.875053:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26427392=0x1934000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760761.875100:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760761.875103:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26431488 00000020:00010000:1.0:1789760761.875109:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760761.875113:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760761.875116:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760761.875119:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760761.875122:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760761.875129:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760761.875134:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760761.875151:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760761.875614:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 26427392<>26431488 00000100:00100000:0.0:1789760761.875759:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512688000:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760761.922381:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973771 gen 3 00000008:00010000:0.0:1789760761.922417:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760761.922423:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760761.922431:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760761.922450:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760761.922471:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512688000:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760761.922554:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6452, 6452] 00000080:00200000:3.0:1789760761.922566:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.922571:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.922583:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26431488 i_blocks 53255 00000020:00200000:3.0:1789760761.922586:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.922594:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.922621:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26431488, count: 4096 00000080:00200000:3.0:1789760761.922628:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26431488, bytes: 4096 00000080:00200000:3.0:1789760761.922637:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.922644:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.922647:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26431488, 26435584) 0 0 00000080:00200000:3.0:1789760761.922650:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.922674:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6453, 6453] 00020000:00200000:3.0:1789760761.922681:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26431488 chunk: [26431488, 26435584) 26435584, 4096 00020000:00200000:3.0:1789760761.922685:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.922700:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6453, 6453] 00000008:00010000:3.0:1789760761.922715:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.922721:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760761.922732:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26431488, 26435584) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.922745:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26431488=0x1935000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.922805:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.922808:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26435584 00000020:00010000:3.0:1789760761.922814:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.922817:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.922819:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.922822:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.922825:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.922831:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.922836:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.922849:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760761.922967:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 26431488<>26435584 00000100:00100000:2.0:1789760761.923133:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512688128:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760761.960001:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973772 gen 3 00000008:00010000:3.0:1789760761.960037:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:3.0:1789760761.960043:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760761.960052:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760761.960075:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:3.0:1789760761.960098:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512688128:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760761.960258:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6453, 6453] 00000080:00200000:0.0:1789760761.960279:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760761.960284:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760761.960295:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26435584 i_blocks 53255 00000020:00200000:0.0:1789760761.960299:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760761.960307:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760761.960331:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26435584, count: 4096 00000080:00200000:0.0:1789760761.960338:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26435584, bytes: 4096 00000080:00200000:0.0:1789760761.960346:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760761.960353:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760761.960355:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26435584, 26439680) 0 0 00000080:00200000:0.0:1789760761.960358:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760761.960371:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6454, 6454] 00020000:00200000:0.0:1789760761.960377:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26435584 chunk: [26435584, 26439680) 26439680, 4096 00020000:00200000:0.0:1789760761.960381:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760761.960396:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6454, 6454] 00000008:00010000:0.0:1789760761.960410:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760761.960415:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760761.960424:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26435584, 26439680) append 0 append lockpos 0 00000080:00200000:0.0:1789760761.960434:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26435584=0x1936000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760761.960480:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760761.960483:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26439680 00000020:00010000:0.0:1789760761.960490:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760761.960493:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.960496:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760761.960499:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760761.960502:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760761.960508:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760761.960513:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760761.960527:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccf00 00000008:00100000:0.0:1789760761.960605:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b100 - object 0x240000400:5 offset 26435584<>26439680 00000100:00100000:0.0:1789760761.960648:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760761.960731:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512688256:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760761.995338:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973773 gen 3 00000008:00010000:0.0:1789760761.995374:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:0.0:1789760761.995381:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760761.995390:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760761.995412:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:0.0:1789760761.995435:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512688256:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760761.995531:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6454, 6454] 00000080:00200000:3.0:1789760761.995545:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760761.995550:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760761.995563:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26439680 i_blocks 53255 00000020:00200000:3.0:1789760761.995567:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760761.995576:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760761.995615:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26439680, count: 4096 00000080:00200000:3.0:1789760761.995622:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26439680, bytes: 4096 00000080:00200000:3.0:1789760761.995631:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760761.995638:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760761.995640:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26439680, 26443776) 0 0 00000080:00200000:3.0:1789760761.995644:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760761.995658:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6455, 6455] 00020000:00200000:3.0:1789760761.995675:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26439680 chunk: [26439680, 26443776) 26443776, 4096 00020000:00200000:3.0:1789760761.995680:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760761.995697:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6455, 6455] 00000008:00010000:3.0:1789760761.995712:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760761.995717:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760761.995726:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26439680, 26443776) append 0 append lockpos 0 00000080:00200000:3.0:1789760761.995737:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26439680=0x1937000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760761.995780:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760761.995782:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26443776 00000020:00010000:3.0:1789760761.995800:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760761.995804:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.995807:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760761.995811:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760761.995814:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760761.995820:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760761.995826:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760761.995841:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760761.995968:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 26439680<>26443776 00000100:00100000:3.0:1789760761.996008:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760761.996022:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512688384:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760762.036924:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973774 gen 3 00000008:00010000:3.0:1789760762.036967:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:3.0:1789760762.036974:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760762.036982:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760762.036999:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:3.0:1789760762.037019:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512688384:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760762.037104:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6455, 6455] 00000080:00200000:2.0:1789760762.037117:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760762.037122:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760762.037133:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26443776 i_blocks 53255 00000020:00200000:2.0:1789760762.037137:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760762.037156:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760762.037184:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26443776, count: 4096 00000080:00200000:2.0:1789760762.037190:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26443776, bytes: 4096 00000080:00200000:2.0:1789760762.037197:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760762.037204:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760762.037205:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26443776, 26447872) 0 0 00000080:00200000:2.0:1789760762.037209:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760762.037221:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6456, 6456] 00020000:00200000:2.0:1789760762.037228:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26443776 chunk: [26443776, 26447872) 26447872, 4096 00020000:00200000:2.0:1789760762.037232:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760762.037246:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6456, 6456] 00000008:00010000:2.0:1789760762.037259:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760762.037264:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760762.037281:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26443776, 26447872) append 0 append lockpos 0 00000080:00200000:2.0:1789760762.037294:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26443776=0x1938000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760762.037336:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760762.037339:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26447872 00000020:00010000:2.0:1789760762.037345:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760762.037349:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.037351:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.037354:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760762.037357:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760762.037363:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760762.037367:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760762.037380:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760762.037455:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 26443776<>26447872 00000100:00100000:2.0:1789760762.037493:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760762.037505:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512688512:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760762.075455:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760762.075467:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760762.075476:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760762.075496:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760762.075522:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512688512:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760762.075940:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6456, 6456] 00000080:00200000:2.0:1789760762.075963:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760762.075969:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760762.075980:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26447872 i_blocks 53255 00000020:00200000:2.0:1789760762.075983:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760762.075991:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760762.076019:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26447872, count: 4096 00000080:00200000:2.0:1789760762.076026:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26447872, bytes: 4096 00000080:00200000:2.0:1789760762.076035:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760762.076042:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760762.076044:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26447872, 26451968) 0 0 00000080:00200000:2.0:1789760762.076048:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760762.076061:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6457, 6457] 00020000:00200000:2.0:1789760762.076068:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26447872 chunk: [26447872, 26451968) 26451968, 4096 00020000:00200000:2.0:1789760762.076073:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760762.076090:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6457, 6457] 00000008:00010000:2.0:1789760762.076105:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760762.076110:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760762.076119:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26447872, 26451968) append 0 append lockpos 0 00000080:00200000:2.0:1789760762.076131:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26447872=0x1939000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760762.076193:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760762.076196:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26451968 00000020:00010000:2.0:1789760762.076202:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760762.076206:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.076209:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.076212:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760762.076215:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760762.076221:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760762.076227:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760762.076242:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760762.076331:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 26447872<>26451968 00000100:00100000:2.0:1789760762.076371:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760762.076386:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512688640:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760762.107526:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973775 gen 3 00000100:00100000:3.0:1789760762.107561:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368c000 x1876699512688640/t4294973776(4294973776) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760778 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760762.107597:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760762.107603:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760762.107611:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760762.107632:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760762.107653:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512688640:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760762.107790:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6457, 6457] 00000080:00200000:2.0:1789760762.107803:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760762.107808:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760762.107819:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26451968 i_blocks 53255 00000020:00200000:2.0:1789760762.107822:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760762.107830:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760762.107857:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26451968, count: 4096 00000080:00200000:2.0:1789760762.107863:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26451968, bytes: 4096 00000080:00200000:2.0:1789760762.107871:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760762.107878:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760762.107880:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26451968, 26456064) 0 0 00000080:00200000:2.0:1789760762.107883:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760762.107897:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6458, 6458] 00020000:00200000:2.0:1789760762.107930:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26451968 chunk: [26451968, 26456064) 26456064, 4096 00020000:00200000:2.0:1789760762.107934:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760762.107950:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6458, 6458] 00000008:00010000:2.0:1789760762.107963:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760762.107968:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760762.107978:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26451968, 26456064) append 0 append lockpos 0 00000080:00200000:2.0:1789760762.107989:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26451968=0x193a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760762.108032:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760762.108034:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26456064 00000020:00010000:2.0:1789760762.108041:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760762.108044:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.108048:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.108050:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760762.108052:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760762.108058:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760762.108064:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760762.108077:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760762.108166:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 26451968<>26456064 00000100:00100000:3.0:1789760762.108317:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512688768:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760762.148441:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973776 gen 3 00000100:00100000:3.0:1789760762.148470:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368ea00 x1876699512688768/t4294973777(4294973777) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760778 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760762.148507:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760762.148513:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760762.148520:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760762.148538:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760762.148557:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512688768:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760762.148893:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6458, 6458] 00000080:00200000:2.0:1789760762.148926:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760762.148931:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760762.148952:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26456064 i_blocks 53255 00000020:00200000:2.0:1789760762.148956:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760762.148964:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760762.148987:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26456064, count: 4096 00000080:00200000:2.0:1789760762.148993:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26456064, bytes: 4096 00000080:00200000:2.0:1789760762.149002:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760762.149009:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760762.149011:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26456064, 26460160) 0 0 00000080:00200000:2.0:1789760762.149014:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760762.149026:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6459, 6459] 00020000:00200000:2.0:1789760762.149033:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26456064 chunk: [26456064, 26460160) 26460160, 4096 00020000:00200000:2.0:1789760762.149037:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760762.149052:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6459, 6459] 00000008:00010000:2.0:1789760762.149065:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760762.149069:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760762.149078:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26456064, 26460160) append 0 append lockpos 0 00000080:00200000:2.0:1789760762.149088:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26456064=0x193b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760762.149130:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760762.149133:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26460160 00000020:00010000:2.0:1789760762.149139:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760762.149262:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.149266:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.149293:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760762.149296:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760762.149304:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760762.149310:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760762.149324:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760762.149402:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 26456064<>26460160 00000100:00100000:3.0:1789760762.149632:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760762.149649:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512688896:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760762.202123:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973778 gen 3 00000008:00010000:3.0:1789760762.202170:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760762.202176:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760762.202193:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760762.202208:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760762.202230:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512688896:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760762.202314:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6459, 6459] 00000080:00200000:2.0:1789760762.202333:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760762.202338:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760762.202349:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26460160 i_blocks 53255 00000020:00200000:2.0:1789760762.202352:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760762.202360:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760762.202393:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26460160, count: 4096 00000080:00200000:2.0:1789760762.202399:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26460160, bytes: 4096 00000080:00200000:2.0:1789760762.202406:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760762.202413:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760762.202415:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26460160, 26464256) 0 0 00000080:00200000:2.0:1789760762.202418:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760762.202430:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6460, 6460] 00020000:00200000:2.0:1789760762.202437:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26460160 chunk: [26460160, 26464256) 26464256, 4096 00020000:00200000:2.0:1789760762.202441:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760762.202456:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6460, 6460] 00000008:00010000:2.0:1789760762.202469:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760762.202474:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760762.202483:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26460160, 26464256) append 0 append lockpos 0 00000080:00200000:2.0:1789760762.202493:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26460160=0x193c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760762.202536:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760762.202538:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26464256 00000020:00010000:2.0:1789760762.202544:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760762.202547:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.202550:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.202553:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760762.202556:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760762.202561:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760762.202566:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760762.202579:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760762.202650:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 26460160<>26464256 00000100:00100000:3.0:1789760762.202829:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512689024:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760762.246567:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973779 gen 3 00000008:00010000:3.0:1789760762.246606:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760762.246614:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760762.246622:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760762.246644:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760762.246680:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512689024:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760762.246877:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6460, 6460] 00000080:00200000:2.0:1789760762.246889:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760762.246894:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760762.246930:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26464256 i_blocks 53255 00000020:00200000:2.0:1789760762.246934:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760762.246952:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760762.246980:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26464256, count: 4096 00000080:00200000:2.0:1789760762.246987:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26464256, bytes: 4096 00000080:00200000:2.0:1789760762.246995:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760762.247002:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760762.247004:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26464256, 26468352) 0 0 00000080:00200000:2.0:1789760762.247008:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760762.247021:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6461, 6461] 00020000:00200000:2.0:1789760762.247027:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26464256 chunk: [26464256, 26468352) 26468352, 4096 00020000:00200000:2.0:1789760762.247031:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760762.247047:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6461, 6461] 00000008:00010000:2.0:1789760762.247060:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760762.247064:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760762.247074:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26464256, 26468352) append 0 append lockpos 0 00000080:00200000:2.0:1789760762.247085:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26464256=0x193d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760762.247128:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760762.247131:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26468352 00000020:00010000:2.0:1789760762.247138:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760762.247141:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.247144:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.247147:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760762.247150:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760762.247156:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760762.247163:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760762.247177:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760762.247257:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 26464256<>26468352 00000100:00100000:3.0:1789760762.247412:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760762.247426:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512689152:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760762.284058:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760762.284069:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760762.284078:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760762.284100:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760762.284125:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512689152:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760762.284212:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6461, 6461] 00000080:00200000:2.0:1789760762.284225:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760762.284231:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760762.284243:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26468352 i_blocks 53255 00000020:00200000:2.0:1789760762.284247:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760762.284256:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760762.284296:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26468352, count: 4096 00000080:00200000:2.0:1789760762.284303:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26468352, bytes: 4096 00000080:00200000:2.0:1789760762.284312:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760762.284319:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760762.284321:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26468352, 26472448) 0 0 00000080:00200000:2.0:1789760762.284324:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760762.284340:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6462, 6462] 00020000:00200000:2.0:1789760762.284348:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26468352 chunk: [26468352, 26472448) 26472448, 4096 00020000:00200000:2.0:1789760762.284351:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760762.284368:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6462, 6462] 00000008:00010000:2.0:1789760762.284383:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760762.284389:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760762.284399:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26468352, 26472448) append 0 append lockpos 0 00000080:00200000:2.0:1789760762.284412:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26468352=0x193e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760762.284461:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760762.284464:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26472448 00000020:00010000:2.0:1789760762.284471:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760762.284475:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.284477:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.284481:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760762.284484:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760762.284489:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760762.284495:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760762.284510:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760762.284586:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 26468352<>26472448 00000100:00100000:3.0:1789760762.284689:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512689280:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760762.318251:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973780 gen 3 00000100:00100000:3.0:1789760762.318291:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368ed80 x1876699512689280/t4294973781(4294973781) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760778 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760762.318331:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760762.318338:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760762.318346:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760762.318363:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760762.318384:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512689280:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760762.318479:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6462, 6462] 00000080:00200000:2.0:1789760762.318493:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760762.318498:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760762.318510:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26472448 i_blocks 53255 00000020:00200000:2.0:1789760762.318514:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760762.318522:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760762.318547:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26472448, count: 4096 00000080:00200000:2.0:1789760762.318553:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26472448, bytes: 4096 00000080:00200000:2.0:1789760762.318561:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760762.318568:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760762.318570:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26472448, 26476544) 0 0 00000080:00200000:2.0:1789760762.318573:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760762.318586:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6463, 6463] 00020000:00200000:2.0:1789760762.318593:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26472448 chunk: [26472448, 26476544) 26476544, 4096 00020000:00200000:2.0:1789760762.318597:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760762.318614:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6463, 6463] 00000008:00010000:2.0:1789760762.318629:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760762.318634:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760762.318644:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26472448, 26476544) append 0 append lockpos 0 00000080:00200000:2.0:1789760762.318655:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26472448=0x193f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760762.318775:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760762.318778:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26476544 00000020:00010000:2.0:1789760762.318784:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760762.318788:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.318790:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.318793:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760762.318796:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760762.318801:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760762.318807:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760762.318820:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760762.318898:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 26472448<>26476544 00000100:00100000:3.0:1789760762.319058:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512689408:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760762.351504:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973782 gen 3 00000008:00010000:3.0:1789760762.351550:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760762.351556:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760762.351564:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760762.351580:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760762.351600:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512689408:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760762.351680:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6463, 6463] 00000080:00200000:2.0:1789760762.351692:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760762.351697:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760762.351708:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26476544 i_blocks 53255 00000020:00200000:2.0:1789760762.351711:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760762.351719:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760762.351744:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26476544, count: 4096 00000080:00200000:2.0:1789760762.351749:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26476544, bytes: 4096 00000080:00200000:2.0:1789760762.351756:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760762.351763:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760762.351765:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26476544, 26480640) 0 0 00000080:00200000:2.0:1789760762.351769:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760762.351782:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6464, 6464] 00020000:00200000:2.0:1789760762.351882:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26476544 chunk: [26476544, 26480640) 26480640, 4096 00020000:00200000:2.0:1789760762.351887:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760762.351928:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6464, 6464] 00000008:00010000:2.0:1789760762.351950:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760762.351955:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760762.351964:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26476544, 26480640) append 0 append lockpos 0 00000080:00200000:2.0:1789760762.351975:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26476544=0x1940000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760762.352018:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760762.352021:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26480640 00000020:00010000:2.0:1789760762.352027:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760762.352030:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.352033:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.352036:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760762.352039:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760762.352045:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760762.352050:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760762.352064:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760762.352141:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 26476544<>26480640 00000100:00100000:3.0:1789760762.352231:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512689536:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760762.385171:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760762.385183:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760762.385191:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760762.385261:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760762.385296:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512689536:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760762.386178:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6464, 6464] 00000080:00200000:2.0:1789760762.386239:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760762.386245:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760762.386256:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26480640 i_blocks 53255 00000020:00200000:2.0:1789760762.386259:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760762.386278:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760762.386301:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26480640, count: 4096 00000080:00200000:2.0:1789760762.386307:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26480640, bytes: 4096 00000080:00200000:2.0:1789760762.386315:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760762.386321:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760762.386323:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26480640, 26484736) 0 0 00000080:00200000:2.0:1789760762.386327:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760762.386339:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6465, 6465] 00020000:00200000:2.0:1789760762.386345:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26480640 chunk: [26480640, 26484736) 26484736, 4096 00020000:00200000:2.0:1789760762.386348:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760762.386361:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6465, 6465] 00000008:00010000:2.0:1789760762.386373:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760762.386377:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760762.386386:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26480640, 26484736) append 0 append lockpos 0 00000080:00200000:2.0:1789760762.386396:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26480640=0x1941000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760762.386438:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760762.386440:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26484736 00000020:00010000:2.0:1789760762.386446:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760762.386449:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.386452:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760762.386455:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760762.386803:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760762.386812:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760762.386818:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760762.386839:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760762.387077:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 26480640<>26484736 00000100:00100000:3.0:1789760762.387126:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512689664:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760762.411786:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973783 gen 3 00000100:00100000:3.0:1789760762.411815:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368df80 x1876699512689664/t4294973784(4294973784) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760778 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760762.412048:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760762.412055:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760762.412062:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760762.412082:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760762.412103:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512689664:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760762.412151:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6465, 6465] 00000080:00200000:0.0:1789760762.412175:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760762.412183:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760762.412195:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26484736 i_blocks 53255 00000020:00200000:0.0:1789760762.412199:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760762.412207:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760762.412233:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26484736, count: 4096 00000080:00200000:0.0:1789760762.412239:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26484736, bytes: 4096 00000080:00200000:0.0:1789760762.412248:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760762.412255:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760762.412257:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26484736, 26488832) 0 0 00000080:00200000:0.0:1789760762.412261:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760762.412404:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6466, 6466] 00020000:00200000:0.0:1789760762.412411:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26484736 chunk: [26484736, 26488832) 26488832, 4096 00020000:00200000:0.0:1789760762.412416:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760762.412433:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6466, 6466] 00000008:00010000:0.0:1789760762.412447:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760762.412451:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760762.412459:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26484736, 26488832) append 0 append lockpos 0 00000080:00200000:0.0:1789760762.412470:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26484736=0x1942000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760762.412516:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760762.412519:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26488832 00000020:00010000:0.0:1789760762.412525:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760762.412528:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.412530:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.412533:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760762.412535:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760762.412542:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760762.412548:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760762.412559:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccf00 00000008:00100000:0.0:1789760762.413685:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b100 - object 0x240000400:5 offset 26484736<>26488832 00000100:00100000:1.0:1789760762.413770:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512689792:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760762.488424:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973785 gen 3 00000008:00010000:1.0:1789760762.488606:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760762.488622:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760762.488631:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760762.488652:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760762.488674:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512689792:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760762.489477:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6466, 6466] 00000080:00200000:0.0:1789760762.489491:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760762.489496:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760762.489508:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26488832 i_blocks 53255 00000020:00200000:0.0:1789760762.489512:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760762.489521:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760762.489549:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26488832, count: 4096 00000080:00200000:0.0:1789760762.489556:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26488832, bytes: 4096 00000080:00200000:0.0:1789760762.489564:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760762.489571:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760762.489573:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26488832, 26492928) 0 0 00000080:00200000:0.0:1789760762.489576:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760762.489590:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6467, 6467] 00020000:00200000:0.0:1789760762.489684:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26488832 chunk: [26488832, 26492928) 26492928, 4096 00020000:00200000:0.0:1789760762.489689:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760762.489707:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6467, 6467] 00000008:00010000:0.0:1789760762.489721:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760762.489726:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760762.489734:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26488832, 26492928) append 0 append lockpos 0 00000080:00200000:0.0:1789760762.489745:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26488832=0x1943000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760762.489792:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760762.489794:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26492928 00000020:00010000:0.0:1789760762.489801:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760762.489803:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.489806:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.489809:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760762.489811:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760762.489818:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760762.489824:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760762.489840:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc000 00000008:00100000:0.0:1789760762.489969:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514bb80 - object 0x240000400:5 offset 26488832<>26492928 00000100:00100000:1.0:1789760762.490610:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512689920:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760762.523745:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760762.523756:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760762.523764:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760762.523784:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760762.523805:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512689920:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760762.524327:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6467, 6467] 00000080:00200000:0.0:1789760762.524341:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760762.524345:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760762.524357:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26492928 i_blocks 53255 00000020:00200000:0.0:1789760762.524360:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760762.524369:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760762.524409:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26492928, count: 4096 00000080:00200000:0.0:1789760762.524415:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26492928, bytes: 4096 00000080:00200000:0.0:1789760762.524424:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760762.524430:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760762.524432:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26492928, 26497024) 0 0 00000080:00200000:0.0:1789760762.524436:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760762.524449:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6468, 6468] 00020000:00200000:0.0:1789760762.524456:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26492928 chunk: [26492928, 26497024) 26497024, 4096 00020000:00200000:0.0:1789760762.524460:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760762.524474:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6468, 6468] 00000008:00010000:0.0:1789760762.524488:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760762.524492:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760762.524502:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26492928, 26497024) append 0 append lockpos 0 00000080:00200000:0.0:1789760762.524512:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26492928=0x1944000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760762.524560:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760762.524563:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26497024 00000020:00010000:0.0:1789760762.524569:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760762.524572:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.524575:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.524578:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760762.524580:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760762.524587:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760762.524592:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760762.524606:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000008:00100000:0.0:1789760762.524685:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149500 - object 0x240000400:5 offset 26492928<>26497024 00000100:00100000:1.0:1789760762.524741:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760762.524755:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512690048:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760762.557339:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973786 gen 3 00000100:00100000:1.0:1789760762.557373:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005149500 x1876699512690048/t4294973787(4294973787) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760778 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760762.557413:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760762.557418:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760762.557426:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760762.557512:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760762.557530:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512690048:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760762.557582:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6468, 6468] 00000080:00200000:0.0:1789760762.557592:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760762.557596:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760762.557606:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26497024 i_blocks 53255 00000020:00200000:0.0:1789760762.557609:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760762.557615:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760762.557635:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26497024, count: 4096 00000080:00200000:0.0:1789760762.557641:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26497024, bytes: 4096 00000080:00200000:0.0:1789760762.557648:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760762.557714:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760762.557717:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26497024, 26501120) 0 0 00000080:00200000:0.0:1789760762.557721:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760762.557734:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6469, 6469] 00020000:00200000:0.0:1789760762.557741:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26497024 chunk: [26497024, 26501120) 26501120, 4096 00020000:00200000:0.0:1789760762.557745:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760762.557760:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6469, 6469] 00000008:00010000:0.0:1789760762.557773:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760762.557778:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760762.557843:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26497024, 26501120) append 0 append lockpos 0 00000080:00200000:0.0:1789760762.557856:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26497024=0x1945000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760762.558031:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760762.558034:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26501120 00000020:00010000:0.0:1789760762.558041:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760762.558044:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.558047:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.558051:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760762.558054:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760762.558061:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760762.558066:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760762.558082:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc780 00000008:00100000:0.0:1789760762.558156:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148000 - object 0x240000400:5 offset 26497024<>26501120 00000100:00100000:1.0:1789760762.558213:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512690176:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760762.624307:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973788 gen 3 00000008:00010000:1.0:1789760762.624349:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760762.624354:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760762.624362:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760762.624381:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760762.624402:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512690176:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760762.624528:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6469, 6469] 00000080:00200000:0.0:1789760762.624540:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760762.624545:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760762.624556:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26501120 i_blocks 53255 00000020:00200000:0.0:1789760762.624559:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760762.624567:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760762.624590:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26501120, count: 4096 00000080:00200000:0.0:1789760762.624596:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26501120, bytes: 4096 00000080:00200000:0.0:1789760762.624603:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760762.624610:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760762.624612:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26501120, 26505216) 0 0 00000080:00200000:0.0:1789760762.624616:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760762.624639:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6470, 6470] 00020000:00200000:0.0:1789760762.624646:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26501120 chunk: [26501120, 26505216) 26505216, 4096 00020000:00200000:0.0:1789760762.624650:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760762.624758:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6470, 6470] 00000008:00010000:0.0:1789760762.624772:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760762.624776:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760762.624785:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26501120, 26505216) append 0 append lockpos 0 00000080:00200000:0.0:1789760762.624806:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26501120=0x1946000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760762.624847:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760762.624849:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26505216 00000020:00010000:0.0:1789760762.624855:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760762.624859:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.624861:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.624864:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760762.624866:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760762.624872:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760762.624877:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760762.624889:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccd80 00000008:00100000:0.0:1789760762.625320:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149f80 - object 0x240000400:5 offset 26501120<>26505216 00000100:00100000:0.0:1789760762.625366:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512690304:192.168.203.156@tcp:4:dd.0 00010000:00010000:0.0:1789760762.654068:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760762.689097:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973789 gen 3 00000008:00010000:0.0:1789760762.689134:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:0.0:1789760762.689141:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760762.689150:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760762.689172:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:0.0:1789760762.689193:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512690304:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760762.689295:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6470, 6470] 00000080:00200000:1.0:1789760762.689306:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760762.689311:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760762.689322:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26505216 i_blocks 53255 00000020:00200000:1.0:1789760762.689326:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760762.689333:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760762.689357:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26505216, count: 4096 00000080:00200000:1.0:1789760762.689363:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26505216, bytes: 4096 00000080:00200000:1.0:1789760762.689370:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760762.689377:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760762.689379:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26505216, 26509312) 0 0 00000080:00200000:1.0:1789760762.689383:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760762.689394:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6471, 6471] 00020000:00200000:1.0:1789760762.689400:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26505216 chunk: [26505216, 26509312) 26509312, 4096 00020000:00200000:1.0:1789760762.689404:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760762.689418:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6471, 6471] 00000008:00010000:1.0:1789760762.689430:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760762.689434:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760762.689442:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26505216, 26509312) append 0 append lockpos 0 00000080:00200000:1.0:1789760762.689453:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26505216=0x1947000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760762.689490:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760762.689492:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26509312 00000020:00010000:1.0:1789760762.689498:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760762.689501:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760762.689504:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760762.689507:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760762.689510:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760762.689516:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760762.689521:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760762.689533:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760762.689598:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95c00 - object 0x240000400:5 offset 26505216<>26509312 00000100:00100000:1.0:1789760762.689636:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760762.689646:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512690432:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760762.771216:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973790 gen 3 00000008:00010000:1.0:1789760762.771255:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760762.771262:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760762.771280:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760762.771298:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760762.771322:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512690432:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760762.771409:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6471, 6471] 00000080:00200000:3.0:1789760762.771423:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.771428:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.771439:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26509312 i_blocks 53255 00000020:00200000:3.0:1789760762.771443:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.771452:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.771481:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26509312, count: 4096 00000080:00200000:3.0:1789760762.771488:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26509312, bytes: 4096 00000080:00200000:3.0:1789760762.771496:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.771504:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.771507:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26509312, 26513408) 0 0 00000080:00200000:3.0:1789760762.771512:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.771527:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6472, 6472] 00020000:00200000:3.0:1789760762.771535:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26509312 chunk: [26509312, 26513408) 26513408, 4096 00020000:00200000:3.0:1789760762.771538:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.771555:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6472, 6472] 00000008:00010000:3.0:1789760762.771568:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.771573:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760762.771583:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26509312, 26513408) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.771621:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26509312=0x1948000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.771669:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.771672:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26513408 00000020:00010000:3.0:1789760762.771678:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.771682:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.771684:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.771688:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.771691:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.771697:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.771703:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.771716:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760762.771797:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 26509312<>26513408 00000100:00100000:3.0:1789760762.771835:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760762.771847:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512690560:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760762.816568:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973791 gen 3 00000008:00010000:3.0:1789760762.816603:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:3.0:1789760762.816608:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760762.816617:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760762.816642:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000080:00200000:1.0:1789760762.816751:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6472, 6472] 00000080:00200000:1.0:1789760762.816765:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760762.816770:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760762.816781:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26513408 i_blocks 53255 00000020:00200000:1.0:1789760762.816784:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760762.816791:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760762.816816:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26513408, count: 4096 00000080:00200000:1.0:1789760762.816823:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26513408, bytes: 4096 00000080:00200000:1.0:1789760762.816831:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760762.816838:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760762.816840:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26513408, 26517504) 0 0 00000080:00200000:1.0:1789760762.816844:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760762.816856:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6473, 6473] 00020000:00200000:1.0:1789760762.816862:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26513408 chunk: [26513408, 26517504) 26517504, 4096 00020000:00200000:1.0:1789760762.816866:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760762.816881:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6473, 6473] 00000008:00010000:1.0:1789760762.816894:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760762.816924:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760762.816934:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26513408, 26517504) append 0 append lockpos 0 00000080:00200000:1.0:1789760762.816956:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26513408=0x1949000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760762.816993:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760762.816995:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26517504 00000020:00010000:1.0:1789760762.817002:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760762.817005:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760762.817008:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760762.817011:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760762.817013:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760762.817019:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760762.817025:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1789760762.817160:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512690560:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760762.817196:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:3.0:1789760762.817349:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 26513408<>26517504 00000100:00100000:2.0:1789760762.817401:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760762.818255:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512690688:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760762.848073:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973792 gen 3 00000008:00010000:3.0:1789760762.848108:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:3.0:1789760762.848115:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760762.848122:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760762.848143:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:3.0:1789760762.848176:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512690688:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760762.848210:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6473, 6473] 00000080:00200000:1.0:1789760762.848226:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760762.848232:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760762.848244:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26517504 i_blocks 53255 00000020:00200000:1.0:1789760762.848247:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760762.848370:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760762.848399:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26517504, count: 4096 00000080:00200000:1.0:1789760762.848408:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26517504, bytes: 4096 00000080:00200000:1.0:1789760762.848421:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760762.848429:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760762.848431:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26517504, 26521600) 0 0 00000080:00200000:1.0:1789760762.848436:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760762.848449:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6474, 6474] 00020000:00200000:1.0:1789760762.848456:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26517504 chunk: [26517504, 26521600) 26521600, 4096 00020000:00200000:1.0:1789760762.848460:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760762.848478:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6474, 6474] 00000008:00010000:1.0:1789760762.848493:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760762.848498:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760762.848507:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26517504, 26521600) append 0 append lockpos 0 00000080:00200000:1.0:1789760762.848524:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26517504=0x194a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760762.848575:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760762.848578:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26521600 00000020:00010000:1.0:1789760762.848585:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760762.848588:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760762.848591:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760762.848594:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760762.848597:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760762.848603:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760762.848609:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760762.848624:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760762.848702:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95c00 - object 0x240000400:5 offset 26517504<>26521600 00000100:00100000:0.0:1789760762.848768:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512690816:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760762.906582:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973793 gen 3 00000008:00010000:0.0:1789760762.906620:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760762.906627:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760762.906635:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760762.906655:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760762.906689:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512690816:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760762.906843:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6474, 6474] 00000080:00200000:3.0:1789760762.906857:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760762.906862:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760762.906873:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26521600 i_blocks 53255 00000020:00200000:3.0:1789760762.906877:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760762.906885:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760762.906955:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26521600, count: 4096 00000080:00200000:3.0:1789760762.906962:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26521600, bytes: 4096 00000080:00200000:3.0:1789760762.906971:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760762.906978:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760762.906980:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26521600, 26525696) 0 0 00000080:00200000:3.0:1789760762.906984:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760762.906997:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6475, 6475] 00020000:00200000:3.0:1789760762.907004:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26521600 chunk: [26521600, 26525696) 26525696, 4096 00020000:00200000:3.0:1789760762.907008:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760762.907024:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6475, 6475] 00000008:00010000:3.0:1789760762.907040:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760762.907046:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760762.907055:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26521600, 26525696) append 0 append lockpos 0 00000080:00200000:3.0:1789760762.907067:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26521600=0x194b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760762.907112:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760762.907115:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26525696 00000020:00010000:3.0:1789760762.907122:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760762.907125:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.907128:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760762.907131:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760762.907134:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760762.907141:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760762.907147:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760762.907163:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760762.907243:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 26521600<>26525696 00000100:00100000:3.0:1789760762.907292:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760762.907306:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512690944:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760762.975364:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:3.0:1789760762.975374:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760762.975383:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760762.975461:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:3.0:1789760762.975485:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512690944:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760762.975552:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6475, 6475] 00000080:00200000:0.0:1789760762.975564:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760762.975569:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760762.975580:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26525696 i_blocks 53255 00000020:00200000:0.0:1789760762.975584:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760762.975592:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760762.975618:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26525696, count: 4096 00000080:00200000:0.0:1789760762.975625:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26525696, bytes: 4096 00000080:00200000:0.0:1789760762.975633:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760762.975639:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760762.975640:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26525696, 26529792) 0 0 00000080:00200000:0.0:1789760762.975644:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760762.975655:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6476, 6476] 00020000:00200000:0.0:1789760762.975662:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26525696 chunk: [26525696, 26529792) 26529792, 4096 00020000:00200000:0.0:1789760762.975666:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760762.975680:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6476, 6476] 00000008:00010000:0.0:1789760762.975693:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760762.975697:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760762.975706:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26525696, 26529792) append 0 append lockpos 0 00000080:00200000:0.0:1789760762.975716:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26525696=0x194c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760762.975758:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760762.975761:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26529792 00000020:00010000:0.0:1789760762.975766:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760762.975770:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.975773:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760762.975776:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760762.975779:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760762.975785:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760762.975852:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760762.975868:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccd80 00000008:00100000:0.0:1789760762.976035:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149f80 - object 0x240000400:5 offset 26525696<>26529792 00000100:00100000:0.0:1789760762.976066:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760762.976081:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512691072:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760763.036809:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973794 gen 3 00000100:00100000:0.0:1789760763.036840:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005149f80 x1876699512691072/t4294973795(4294973795) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760778 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760763.036876:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:0.0:1789760763.036882:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760763.036889:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760763.036991:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:0.0:1789760763.037004:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512691072:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760763.039888:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6476, 6476] 00000080:00200000:3.0:1789760763.039930:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760763.039936:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760763.039975:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26529792 i_blocks 53255 00000020:00200000:3.0:1789760763.039980:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760763.039992:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760763.040016:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26529792, count: 4096 00000080:00200000:3.0:1789760763.040023:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26529792, bytes: 4096 00000080:00200000:3.0:1789760763.040034:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760763.040042:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760763.040044:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26529792, 26533888) 0 0 00000080:00200000:3.0:1789760763.040048:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760763.040063:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6477, 6477] 00020000:00200000:3.0:1789760763.040069:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26529792 chunk: [26529792, 26533888) 26533888, 4096 00020000:00200000:3.0:1789760763.040073:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760763.040088:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6477, 6477] 00000008:00010000:3.0:1789760763.040101:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760763.040106:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760763.040115:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26529792, 26533888) append 0 append lockpos 0 00000080:00200000:3.0:1789760763.040126:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26529792=0x194d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760763.040174:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760763.040177:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26533888 00000020:00010000:3.0:1789760763.040184:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760763.040187:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.040190:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.040193:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760763.040196:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760763.040202:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760763.040207:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760763.040219:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760763.040307:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad180 - object 0x240000400:5 offset 26529792<>26533888 00000100:00100000:2.0:1789760763.040857:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760763.040879:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512691200:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760763.079338:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973795 gen 3 00000100:00100000:3.0:1789760763.079370:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc0107ad180 x1876699512691200/t4294973796(4294973796) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760779 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760763.079408:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:3.0:1789760763.079414:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760763.079422:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760763.079445:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:3.0:1789760763.079466:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512691200:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760763.079562:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6477, 6477] 00000080:00200000:0.0:1789760763.079575:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.079580:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.079591:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26533888 i_blocks 53255 00000020:00200000:0.0:1789760763.079594:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.079602:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.079629:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26533888, count: 4096 00000080:00200000:0.0:1789760763.079635:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26533888, bytes: 4096 00000080:00200000:0.0:1789760763.079642:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.079648:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.079650:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26533888, 26537984) 0 0 00000080:00200000:0.0:1789760763.079662:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.079675:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6478, 6478] 00020000:00200000:0.0:1789760763.079681:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26533888 chunk: [26533888, 26537984) 26537984, 4096 00020000:00200000:0.0:1789760763.079684:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.079697:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6478, 6478] 00000008:00010000:0.0:1789760763.079709:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.079714:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760763.079722:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26533888, 26537984) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.079733:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26533888=0x194e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.079773:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.079776:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26537984 00000020:00010000:0.0:1789760763.079782:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.079785:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.079800:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.079803:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.079805:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.079811:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.079818:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.079832:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccd80 00000008:00100000:0.0:1789760763.080198:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149180 - object 0x240000400:5 offset 26533888<>26537984 00000100:00100000:1.0:1789760763.080384:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512691328:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760763.115719:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973797 gen 3 00000008:00010000:1.0:1789760763.115767:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760763.115774:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.115782:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.115862:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760763.115889:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512691328:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760763.117324:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6478, 6478] 00000080:00200000:0.0:1789760763.117342:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.117347:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.117360:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26537984 i_blocks 53255 00000020:00200000:0.0:1789760763.117364:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.117373:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.117401:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26537984, count: 4096 00000080:00200000:0.0:1789760763.117408:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26537984, bytes: 4096 00000080:00200000:0.0:1789760763.117419:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.117428:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.117430:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26537984, 26542080) 0 0 00000080:00200000:0.0:1789760763.117435:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.117449:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6479, 6479] 00020000:00200000:0.0:1789760763.117457:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26537984 chunk: [26537984, 26542080) 26542080, 4096 00020000:00200000:0.0:1789760763.117462:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.117479:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6479, 6479] 00000008:00010000:0.0:1789760763.117493:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.117498:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760763.117507:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26537984, 26542080) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.117519:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26537984=0x194f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.117566:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.117568:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26542080 00000020:00010000:0.0:1789760763.117575:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.117579:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.117581:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.117584:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.117587:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.117594:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.117599:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.117623:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc180 00000008:00100000:0.0:1789760763.117854:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149c00 - object 0x240000400:5 offset 26537984<>26542080 00000100:00100000:1.0:1789760763.118173:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512691456:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760763.156949:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973798 gen 3 00000008:00010000:1.0:1789760763.156986:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760763.156993:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.157002:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.157023:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760763.157046:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512691456:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760763.157197:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6479, 6479] 00000080:00200000:0.0:1789760763.157212:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.157216:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.157228:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26542080 i_blocks 53255 00000020:00200000:0.0:1789760763.157231:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.157239:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.157277:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26542080, count: 4096 00000080:00200000:0.0:1789760763.157283:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26542080, bytes: 4096 00000080:00200000:0.0:1789760763.157290:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.157297:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.157299:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26542080, 26546176) 0 0 00000080:00200000:0.0:1789760763.157303:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.157316:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6480, 6480] 00020000:00200000:0.0:1789760763.157322:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26542080 chunk: [26542080, 26546176) 26546176, 4096 00020000:00200000:0.0:1789760763.157326:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.157342:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6480, 6480] 00000008:00010000:0.0:1789760763.157355:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.157360:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760763.157370:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26542080, 26546176) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.157380:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26542080=0x1950000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.157426:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.157429:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26546176 00000020:00010000:0.0:1789760763.157436:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.157438:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.157441:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.157444:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.157447:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.157453:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.157460:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.157475:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000008:00100000:0.0:1789760763.157560:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b800 - object 0x240000400:5 offset 26542080<>26546176 00000100:00100000:1.0:1789760763.157730:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512691584:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760763.189839:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973799 gen 3 00000008:00010000:1.0:1789760763.189875:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760763.189881:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.189890:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.189929:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760763.189951:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512691584:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760763.190061:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6480, 6480] 00000080:00200000:0.0:1789760763.190074:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.190079:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.190092:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26546176 i_blocks 53255 00000020:00200000:0.0:1789760763.190095:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.190104:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.190131:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26546176, count: 4096 00000080:00200000:0.0:1789760763.190137:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26546176, bytes: 4096 00000080:00200000:0.0:1789760763.190145:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.190152:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.190154:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26546176, 26550272) 0 0 00000080:00200000:0.0:1789760763.190157:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.190171:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6481, 6481] 00020000:00200000:0.0:1789760763.190178:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26546176 chunk: [26546176, 26550272) 26550272, 4096 00020000:00200000:0.0:1789760763.190181:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.190197:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6481, 6481] 00000008:00010000:0.0:1789760763.190211:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.190216:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760763.190225:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26546176, 26550272) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.190235:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26546176=0x1951000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.190289:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.190292:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26550272 00000020:00010000:0.0:1789760763.190298:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.190301:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.190304:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.190306:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.190309:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.190314:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.190319:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.190333:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccc00 00000008:00100000:0.0:1789760763.190403:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148380 - object 0x240000400:5 offset 26546176<>26550272 00000100:00100000:1.0:1789760763.190464:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512691712:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760763.229212:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973800 gen 3 00000008:00010000:1.0:1789760763.229245:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760763.229251:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.229271:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.229348:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760763.229373:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512691712:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760763.229448:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6481, 6481] 00000080:00200000:0.0:1789760763.229462:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.229467:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.229478:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26550272 i_blocks 53255 00000020:00200000:0.0:1789760763.229481:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.229489:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.229516:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26550272, count: 4096 00000080:00200000:0.0:1789760763.229522:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26550272, bytes: 4096 00000080:00200000:0.0:1789760763.229529:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.229536:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.229539:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26550272, 26554368) 0 0 00000080:00200000:0.0:1789760763.229542:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.229556:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6482, 6482] 00020000:00200000:0.0:1789760763.229563:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26550272 chunk: [26550272, 26554368) 26554368, 4096 00020000:00200000:0.0:1789760763.229567:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.229582:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6482, 6482] 00000008:00010000:0.0:1789760763.229595:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.229600:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760763.229608:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26550272, 26554368) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.229619:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26550272=0x1952000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.229781:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.229785:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26554368 00000020:00010000:0.0:1789760763.229854:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.229858:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.229861:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.229864:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.229866:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.229873:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.229879:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.229894:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cce40 00000008:00100000:0.0:1789760763.230165:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b480 - object 0x240000400:5 offset 26550272<>26554368 00000100:00100000:1.0:1789760763.230214:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760763.230282:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512691840:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760763.265183:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973801 gen 3 00000008:00010000:1.0:1789760763.265221:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760763.265228:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.265236:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.265259:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760763.265291:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512691840:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760763.265382:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6482, 6482] 00000080:00200000:3.0:1789760763.265393:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760763.265399:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760763.265409:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26554368 i_blocks 53255 00000020:00200000:3.0:1789760763.265413:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760763.265421:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760763.265445:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26554368, count: 4096 00000080:00200000:3.0:1789760763.265451:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26554368, bytes: 4096 00000080:00200000:3.0:1789760763.265459:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760763.265466:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760763.265468:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26554368, 26558464) 0 0 00000080:00200000:3.0:1789760763.265472:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760763.265484:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6483, 6483] 00020000:00200000:3.0:1789760763.265490:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26554368 chunk: [26554368, 26558464) 26558464, 4096 00020000:00200000:3.0:1789760763.265494:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760763.265509:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6483, 6483] 00000008:00010000:3.0:1789760763.265522:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760763.265526:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760763.265535:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26554368, 26558464) append 0 append lockpos 0 00000080:00200000:3.0:1789760763.265546:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26554368=0x1953000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760763.265590:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760763.265593:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26558464 00000020:00010000:3.0:1789760763.265599:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760763.265602:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.265605:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.265608:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760763.265610:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760763.265616:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760763.265622:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760763.265636:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760763.265725:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac380 - object 0x240000400:5 offset 26554368<>26558464 00000100:00100000:2.0:1789760763.265940:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512691968:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760763.299732:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:3.0:1789760763.299744:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760763.299754:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760763.299777:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:3.0:1789760763.299810:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512691968:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760763.299923:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6483, 6483] 00000080:00200000:1.0:1789760763.299942:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760763.299947:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760763.299958:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26558464 i_blocks 53255 00000020:00200000:1.0:1789760763.299961:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760763.299969:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760763.299995:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26558464, count: 4096 00000080:00200000:1.0:1789760763.300000:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26558464, bytes: 4096 00000080:00200000:1.0:1789760763.300008:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760763.300015:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760763.300017:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26558464, 26562560) 0 0 00000080:00200000:1.0:1789760763.300021:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760763.300034:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6484, 6484] 00020000:00200000:1.0:1789760763.300041:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26558464 chunk: [26558464, 26562560) 26562560, 4096 00020000:00200000:1.0:1789760763.300044:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760763.300060:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6484, 6484] 00000008:00010000:1.0:1789760763.300076:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760763.300081:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760763.300091:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26558464, 26562560) append 0 append lockpos 0 00000080:00200000:1.0:1789760763.300101:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26558464=0x1954000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760763.300144:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760763.300147:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26562560 00000020:00010000:1.0:1789760763.300158:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760763.300161:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760763.300163:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760763.300166:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760763.300168:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760763.300174:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760763.300179:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760763.300192:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760763.300279:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 26558464<>26562560 00000100:00100000:0.0:1789760763.300375:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760763.300390:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512692096:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760763.336111:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973803 gen 3 00000008:00010000:0.0:1789760763.336159:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760763.336166:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760763.336174:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760763.336195:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760763.336218:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512692096:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760763.336411:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6484, 6484] 00000080:00200000:1.0:1789760763.336424:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760763.336430:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760763.336443:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26562560 i_blocks 53255 00000020:00200000:1.0:1789760763.336446:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760763.336454:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760763.336480:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26562560, count: 4096 00000080:00200000:1.0:1789760763.336486:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26562560, bytes: 4096 00000080:00200000:1.0:1789760763.336494:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760763.336502:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760763.336504:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26562560, 26566656) 0 0 00000080:00200000:1.0:1789760763.336507:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760763.336520:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6485, 6485] 00020000:00200000:1.0:1789760763.336526:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26562560 chunk: [26562560, 26566656) 26566656, 4096 00020000:00200000:1.0:1789760763.336531:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760763.336546:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6485, 6485] 00000008:00010000:1.0:1789760763.336561:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760763.336565:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760763.336574:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26562560, 26566656) append 0 append lockpos 0 00000080:00200000:1.0:1789760763.336585:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26562560=0x1955000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760763.336626:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760763.336629:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26566656 00000020:00010000:1.0:1789760763.336635:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760763.336638:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760763.336641:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760763.336645:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760763.336648:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760763.336662:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760763.336669:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760763.336682:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760763.336753:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 26562560<>26566656 00000100:00100000:0.0:1789760763.336898:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512692224:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760763.369183:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973804 gen 3 00000008:00010000:0.0:1789760763.369220:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760763.369227:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760763.369235:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760763.369270:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760763.369295:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512692224:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760763.369408:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6485, 6485] 00000080:00200000:1.0:1789760763.369421:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760763.369426:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760763.369437:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26566656 i_blocks 53255 00000020:00200000:1.0:1789760763.369440:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760763.369448:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760763.369473:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26566656, count: 4096 00000080:00200000:1.0:1789760763.369480:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26566656, bytes: 4096 00000080:00200000:1.0:1789760763.369488:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760763.369495:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760763.369497:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26566656, 26570752) 0 0 00000080:00200000:1.0:1789760763.369501:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760763.369514:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6486, 6486] 00020000:00200000:1.0:1789760763.369520:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26566656 chunk: [26566656, 26570752) 26570752, 4096 00020000:00200000:1.0:1789760763.369524:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760763.369538:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6486, 6486] 00000008:00010000:1.0:1789760763.369553:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760763.369558:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760763.369567:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26566656, 26570752) append 0 append lockpos 0 00000080:00200000:1.0:1789760763.369578:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26566656=0x1956000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760763.369619:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760763.369621:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26570752 00000020:00010000:1.0:1789760763.369628:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760763.369631:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760763.369634:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760763.369637:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760763.369640:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760763.369646:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760763.369664:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760763.369678:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760763.369755:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94700 - object 0x240000400:5 offset 26566656<>26570752 00000100:00100000:0.0:1789760763.369923:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760763.369937:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512692352:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760763.399138:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973805 gen 3 00000008:00010000:0.0:1789760763.399183:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760763.399190:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760763.399198:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760763.399276:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760763.399300:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512692352:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760763.399457:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6486, 6486] 00000080:00200000:1.0:1789760763.399468:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760763.399472:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760763.399481:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26570752 i_blocks 53255 00000020:00200000:1.0:1789760763.399484:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760763.399491:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760763.399514:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26570752, count: 4096 00000080:00200000:1.0:1789760763.399518:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26570752, bytes: 4096 00000080:00200000:1.0:1789760763.399525:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760763.399530:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760763.399532:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26570752, 26574848) 0 0 00000080:00200000:1.0:1789760763.399535:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760763.399548:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6487, 6487] 00020000:00200000:1.0:1789760763.399554:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26570752 chunk: [26570752, 26574848) 26574848, 4096 00020000:00200000:1.0:1789760763.399557:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760763.399571:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6487, 6487] 00000008:00010000:1.0:1789760763.399584:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760763.399590:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760763.399599:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26570752, 26574848) append 0 append lockpos 0 00000080:00200000:1.0:1789760763.399609:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26570752=0x1957000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760763.399651:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760763.399654:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26574848 00000020:00010000:1.0:1789760763.399662:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760763.399664:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760763.399667:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760763.399670:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760763.399672:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760763.399678:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760763.399684:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760763.399698:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760763.399772:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 26570752<>26574848 00000100:00100000:0.0:1789760763.399828:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512692480:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760763.433017:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973806 gen 3 00000008:00010000:0.0:1789760763.433053:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760763.433060:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760763.433068:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760763.433077:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760763.433104:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512692480:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760763.433350:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6487, 6487] 00000080:00200000:1.0:1789760763.433362:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760763.433366:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760763.433378:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26574848 i_blocks 53255 00000020:00200000:1.0:1789760763.433381:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760763.433389:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760763.433414:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26574848, count: 4096 00000080:00200000:1.0:1789760763.433420:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26574848, bytes: 4096 00000080:00200000:1.0:1789760763.433428:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760763.433435:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760763.433437:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26574848, 26578944) 0 0 00000080:00200000:1.0:1789760763.433441:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760763.433453:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6488, 6488] 00020000:00200000:1.0:1789760763.433459:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26574848 chunk: [26574848, 26578944) 26578944, 4096 00020000:00200000:1.0:1789760763.433463:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760763.433476:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6488, 6488] 00000008:00010000:1.0:1789760763.433487:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760763.433492:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760763.433499:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26574848, 26578944) append 0 append lockpos 0 00000080:00200000:1.0:1789760763.433509:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26574848=0x1958000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760763.433547:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760763.433550:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26578944 00000020:00010000:1.0:1789760763.433556:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760763.433559:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760763.433562:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760763.433565:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760763.433567:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760763.433572:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760763.433577:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760763.433589:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760763.433722:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94380 - object 0x240000400:5 offset 26574848<>26578944 00000100:00100000:0.0:1789760763.433773:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512692608:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760763.469335:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973807 gen 3 00000008:00010000:0.0:1789760763.469383:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760763.469389:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760763.469398:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760763.469415:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760763.469438:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512692608:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760763.469737:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6488, 6488] 00000080:00200000:1.0:1789760763.469751:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760763.469756:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760763.469769:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26578944 i_blocks 53255 00000020:00200000:1.0:1789760763.469772:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760763.469779:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760763.469803:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26578944, count: 4096 00000080:00200000:1.0:1789760763.469810:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26578944, bytes: 4096 00000080:00200000:1.0:1789760763.469818:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760763.469825:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760763.469827:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26578944, 26583040) 0 0 00000080:00200000:1.0:1789760763.469831:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760763.469843:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6489, 6489] 00020000:00200000:1.0:1789760763.469850:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26578944 chunk: [26578944, 26583040) 26583040, 4096 00020000:00200000:1.0:1789760763.469854:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760763.469869:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6489, 6489] 00000008:00010000:1.0:1789760763.469883:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760763.469887:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760763.469897:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26578944, 26583040) append 0 append lockpos 0 00000080:00200000:1.0:1789760763.469935:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26578944=0x1959000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760763.469980:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760763.469983:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26583040 00000020:00010000:1.0:1789760763.469989:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760763.469992:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760763.469995:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760763.469998:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760763.470001:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760763.470007:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760763.470013:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760763.470026:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760763.470096:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 26578944<>26583040 00000100:00100000:0.0:1789760763.470341:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512692736:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760763.516283:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973808 gen 3 00000008:00010000:0.0:1789760763.516317:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760763.516323:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760763.516331:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760763.516449:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760763.516471:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512692736:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760763.517314:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6489, 6489] 00000080:00200000:1.0:1789760763.517330:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760763.517335:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760763.517348:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26583040 i_blocks 53255 00000020:00200000:1.0:1789760763.517352:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760763.517361:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760763.517461:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26583040, count: 4096 00000080:00200000:1.0:1789760763.517469:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26583040, bytes: 4096 00000080:00200000:1.0:1789760763.517479:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760763.517487:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760763.517489:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26583040, 26587136) 0 0 00000080:00200000:1.0:1789760763.517493:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760763.517506:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6490, 6490] 00020000:00200000:1.0:1789760763.517513:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26583040 chunk: [26583040, 26587136) 26587136, 4096 00020000:00200000:1.0:1789760763.517517:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760763.517595:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6490, 6490] 00000008:00010000:1.0:1789760763.517611:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760763.517615:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760763.517625:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26583040, 26587136) append 0 append lockpos 0 00000080:00200000:1.0:1789760763.517637:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26583040=0x195a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760763.517860:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760763.517863:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26587136 00000020:00010000:1.0:1789760763.517870:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760763.517873:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760763.517876:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760763.517879:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760763.517882:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760763.517889:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760763.517894:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760763.517999:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760763.518083:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 26583040<>26587136 00000100:00100000:0.0:1789760763.518155:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512692864:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760763.554805:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973809 gen 3 00000008:00010000:0.0:1789760763.554842:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760763.554849:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760763.554857:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760763.554939:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760763.554964:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512692864:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760763.555047:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6490, 6490] 00000080:00200000:3.0:1789760763.555060:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760763.555065:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760763.555077:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26587136 i_blocks 53255 00000020:00200000:3.0:1789760763.555080:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760763.555087:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760763.555114:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26587136, count: 4096 00000080:00200000:3.0:1789760763.555120:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26587136, bytes: 4096 00000080:00200000:3.0:1789760763.555128:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760763.555135:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760763.555137:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26587136, 26591232) 0 0 00000080:00200000:3.0:1789760763.555141:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760763.555488:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6491, 6491] 00020000:00200000:3.0:1789760763.555495:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26587136 chunk: [26587136, 26591232) 26591232, 4096 00020000:00200000:3.0:1789760763.555498:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760763.555513:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6491, 6491] 00000008:00010000:3.0:1789760763.555527:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760763.555733:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760763.555744:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26587136, 26591232) append 0 append lockpos 0 00000080:00200000:3.0:1789760763.555753:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26587136=0x195b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760763.555794:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760763.555797:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26591232 00000020:00010000:3.0:1789760763.555803:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760763.555807:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.555810:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.555812:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760763.555814:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760763.555821:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760763.555826:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760763.555845:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760763.556073:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af480 - object 0x240000400:5 offset 26587136<>26591232 00000100:00100000:2.0:1789760763.556130:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512692992:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760763.604138:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973810 gen 3 00000008:00010000:3.0:1789760763.604178:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:3.0:1789760763.604185:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760763.604194:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760763.604255:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:3.0:1789760763.604289:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512692992:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760763.604460:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6491, 6491] 00000080:00200000:2.0:1789760763.604474:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760763.604480:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760763.604491:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26591232 i_blocks 53255 00000020:00200000:2.0:1789760763.604495:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760763.604503:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760763.604527:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26591232, count: 4096 00000080:00200000:2.0:1789760763.604534:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26591232, bytes: 4096 00000080:00200000:2.0:1789760763.604543:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760763.604551:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760763.604553:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26591232, 26595328) 0 0 00000080:00200000:2.0:1789760763.604557:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760763.604571:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6492, 6492] 00020000:00200000:2.0:1789760763.604713:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26591232 chunk: [26591232, 26595328) 26595328, 4096 00020000:00200000:2.0:1789760763.604720:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760763.604739:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6492, 6492] 00000008:00010000:2.0:1789760763.604753:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760763.604758:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760763.604767:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26591232, 26595328) append 0 append lockpos 0 00000080:00200000:2.0:1789760763.604779:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26591232=0x195c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760763.604826:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760763.604829:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26595328 00000020:00010000:2.0:1789760763.604836:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760763.604839:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760763.604842:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760763.604845:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760763.604849:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760763.604855:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760763.604861:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760763.604876:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760763.605328:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 26591232<>26595328 00000100:00100000:2.0:1789760763.605387:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760763.605403:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512693120:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760763.646586:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760763.646598:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760763.646607:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760763.646629:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760763.646766:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512693120:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760763.647957:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6492, 6492] 00000080:00200000:0.0:1789760763.647971:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.647977:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.647989:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26595328 i_blocks 53255 00000020:00200000:0.0:1789760763.647993:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.648001:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.648026:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26595328, count: 4096 00000080:00200000:0.0:1789760763.648032:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26595328, bytes: 4096 00000080:00200000:0.0:1789760763.648040:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.648047:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.648050:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26595328, 26599424) 0 0 00000080:00200000:0.0:1789760763.648054:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.648068:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6493, 6493] 00020000:00200000:0.0:1789760763.648075:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26595328 chunk: [26595328, 26599424) 26599424, 4096 00020000:00200000:0.0:1789760763.648079:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.648094:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6493, 6493] 00000008:00010000:0.0:1789760763.648106:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.648111:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760763.648120:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26595328, 26599424) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.648130:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26595328=0x195d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.648172:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.648175:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26599424 00000020:00010000:0.0:1789760763.648182:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.648185:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.648188:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.648191:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.648194:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.648201:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.648207:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.648222:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cce40 00000008:00100000:0.0:1789760763.648309:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149880 - object 0x240000400:5 offset 26595328<>26599424 00000100:00100000:1.0:1789760763.648393:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512693248:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760763.677529:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760763.700775:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973812 gen 3 00000008:00010000:1.0:1789760763.700825:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760763.700831:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.700840:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.700860:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760763.700887:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512693248:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760763.701021:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6493, 6493] 00000080:00200000:0.0:1789760763.701034:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.701039:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.701050:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26599424 i_blocks 53255 00000020:00200000:0.0:1789760763.701053:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.701062:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.701093:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26599424, count: 4096 00000080:00200000:0.0:1789760763.701099:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26599424, bytes: 4096 00000080:00200000:0.0:1789760763.701108:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.701115:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.701118:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26599424, 26603520) 0 0 00000080:00200000:0.0:1789760763.701121:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.701134:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6494, 6494] 00020000:00200000:0.0:1789760763.701140:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26599424 chunk: [26599424, 26603520) 26603520, 4096 00020000:00200000:0.0:1789760763.701144:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.701158:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6494, 6494] 00000008:00010000:0.0:1789760763.701170:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.701175:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760763.701183:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26599424, 26603520) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.701192:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26599424=0x195e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.701232:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.701235:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26603520 00000020:00010000:0.0:1789760763.701241:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.701244:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.701246:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.701249:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.701253:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.701269:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.701275:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.701289:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccc00 00000008:00100000:0.0:1789760763.701359:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a300 - object 0x240000400:5 offset 26599424<>26603520 00000100:00100000:1.0:1789760763.702403:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760763.702424:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512693376:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760763.761897:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973813 gen 3 00000008:00010000:1.0:1789760763.761959:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760763.761965:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.761973:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.761996:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760763.762016:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512693376:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760763.762116:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6494, 6494] 00000080:00200000:0.0:1789760763.762129:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.762134:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.762146:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26603520 i_blocks 53255 00000020:00200000:0.0:1789760763.762158:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.762167:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.762192:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26603520, count: 4096 00000080:00200000:0.0:1789760763.762198:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26603520, bytes: 4096 00000080:00200000:0.0:1789760763.762206:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.762212:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.762215:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26603520, 26607616) 0 0 00000080:00200000:0.0:1789760763.762219:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.762232:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6495, 6495] 00020000:00200000:0.0:1789760763.762238:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26603520 chunk: [26603520, 26607616) 26607616, 4096 00020000:00200000:0.0:1789760763.762241:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.762255:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6495, 6495] 00000008:00010000:0.0:1789760763.762279:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.762284:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760763.762293:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26603520, 26607616) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.762304:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26603520=0x195f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.762348:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.762351:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26607616 00000020:00010000:0.0:1789760763.762357:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.762360:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.762714:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.762718:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.762720:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.762729:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.762734:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.762749:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000008:00100000:0.0:1789760763.762834:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a680 - object 0x240000400:5 offset 26603520<>26607616 00000100:00100000:0.0:1789760763.762875:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760763.762888:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512693504:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760763.805988:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:0.0:1789760763.806000:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760763.806009:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760763.806079:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:0.0:1789760763.806106:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512693504:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760763.807971:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6495, 6495] 00000080:00200000:3.0:1789760763.807988:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760763.807993:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760763.808006:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26607616 i_blocks 53255 00000020:00200000:3.0:1789760763.808011:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760763.808023:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760763.808050:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26607616, count: 4096 00000080:00200000:3.0:1789760763.808057:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26607616, bytes: 4096 00000080:00200000:3.0:1789760763.808067:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760763.808074:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760763.808076:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26607616, 26611712) 0 0 00000080:00200000:3.0:1789760763.808081:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760763.808094:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6496, 6496] 00020000:00200000:3.0:1789760763.808102:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26607616 chunk: [26607616, 26611712) 26611712, 4096 00020000:00200000:3.0:1789760763.808105:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760763.808119:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6496, 6496] 00000008:00010000:3.0:1789760763.808134:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760763.808138:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760763.808147:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26607616, 26611712) append 0 append lockpos 0 00000080:00200000:3.0:1789760763.808157:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26607616=0x1960000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760763.808211:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760763.808214:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26611712 00000020:00010000:3.0:1789760763.808221:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760763.808224:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.808226:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760763.808230:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760763.808232:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760763.808238:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760763.808243:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760763.808380:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760763.808467:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af480 - object 0x240000400:5 offset 26607616<>26611712 00000100:00100000:3.0:1789760763.808521:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760763.808540:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512693632:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760763.841522:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973815 gen 3 00000008:00010000:3.0:1789760763.841573:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:3.0:1789760763.841580:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760763.841587:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760763.841609:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:3.0:1789760763.841631:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512693632:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760763.841725:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6496, 6496] 00000080:00200000:0.0:1789760763.841738:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.841743:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.841757:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26611712 i_blocks 53255 00000020:00200000:0.0:1789760763.841761:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.841768:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.841796:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26611712, count: 4096 00000080:00200000:0.0:1789760763.841802:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26611712, bytes: 4096 00000080:00200000:0.0:1789760763.841810:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.841817:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.841819:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26611712, 26615808) 0 0 00000080:00200000:0.0:1789760763.841823:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.841837:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6497, 6497] 00020000:00200000:0.0:1789760763.841843:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26611712 chunk: [26611712, 26615808) 26615808, 4096 00020000:00200000:0.0:1789760763.841847:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.841862:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6497, 6497] 00000008:00010000:0.0:1789760763.841877:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.841881:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760763.841891:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26611712, 26615808) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.841933:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26611712=0x1961000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.841977:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.841980:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26615808 00000020:00010000:0.0:1789760763.841988:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.841991:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.841994:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.841997:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.842000:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.842007:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.842013:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.842027:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000008:00100000:0.0:1789760763.842120:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148e00 - object 0x240000400:5 offset 26611712<>26615808 00000100:00100000:1.0:1789760763.842403:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760763.842420:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512693760:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760763.918715:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973816 gen 3 00000008:00010000:1.0:1789760763.919199:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760763.919205:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.919214:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.919235:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760763.919265:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512693760:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760763.919853:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6497, 6497] 00000080:00200000:0.0:1789760763.919866:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.919872:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.919882:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26615808 i_blocks 53255 00000020:00200000:0.0:1789760763.919886:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.919894:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.919939:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26615808, count: 4096 00000080:00200000:0.0:1789760763.919946:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26615808, bytes: 4096 00000080:00200000:0.0:1789760763.919954:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.919961:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.919963:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26615808, 26619904) 0 0 00000080:00200000:0.0:1789760763.919967:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.919980:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6498, 6498] 00020000:00200000:0.0:1789760763.919987:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26615808 chunk: [26615808, 26619904) 26619904, 4096 00020000:00200000:0.0:1789760763.919991:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.920007:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6498, 6498] 00000008:00010000:0.0:1789760763.920020:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.920024:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760763.920032:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26615808, 26619904) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.920042:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26615808=0x1962000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.920161:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.920164:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26619904 00000020:00010000:0.0:1789760763.920172:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.920175:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.920178:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.920181:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.920183:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.920189:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.920195:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.920208:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc180 00000008:00100000:0.0:1789760763.920294:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148a80 - object 0x240000400:5 offset 26615808<>26619904 00000100:00100000:1.0:1789760763.920599:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512693888:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760763.970513:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973817 gen 3 00000008:00010000:1.0:1789760763.970551:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760763.970558:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760763.970567:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760763.970651:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760763.970675:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512693888:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760763.970918:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6498, 6498] 00000080:00200000:0.0:1789760763.970930:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760763.970935:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760763.970946:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26619904 i_blocks 53255 00000020:00200000:0.0:1789760763.970949:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760763.970955:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760763.970979:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26619904, count: 4096 00000080:00200000:0.0:1789760763.970984:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26619904, bytes: 4096 00000080:00200000:0.0:1789760763.970991:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760763.970998:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760763.971000:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26619904, 26624000) 0 0 00000080:00200000:0.0:1789760763.971004:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760763.971015:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6499, 6499] 00020000:00200000:0.0:1789760763.971021:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26619904 chunk: [26619904, 26624000) 26624000, 4096 00020000:00200000:0.0:1789760763.971024:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760763.971037:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6499, 6499] 00000008:00010000:0.0:1789760763.971049:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760763.971053:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760763.971062:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26619904, 26624000) append 0 append lockpos 0 00000080:00200000:0.0:1789760763.971083:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26619904=0x1963000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760763.971122:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760763.971124:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26624000 00000020:00010000:0.0:1789760763.971130:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760763.971133:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.971136:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760763.971140:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760763.971142:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760763.971149:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760763.971155:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760763.971167:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccd80 00000008:00100000:0.0:1789760763.971265:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514ad80 - object 0x240000400:5 offset 26619904<>26624000 00000100:00100000:1.0:1789760763.971497:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760763.971511:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512694016:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760764.008085:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973818 gen 3 00000008:00010000:1.0:1789760764.008118:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760764.008125:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.008133:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.008153:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760764.008176:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512694016:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760764.008284:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6499, 6499] 00000080:00200000:0.0:1789760764.008297:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760764.008302:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760764.008312:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26624000 i_blocks 53255 00000020:00200000:0.0:1789760764.008316:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760764.008324:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760764.008346:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26624000, count: 4096 00000080:00200000:0.0:1789760764.008351:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26624000, bytes: 4096 00000080:00200000:0.0:1789760764.008358:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760764.008478:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760764.008480:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26624000, 26628096) 0 0 00000080:00200000:0.0:1789760764.008484:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760764.008496:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6500, 6500] 00020000:00200000:0.0:1789760764.008503:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26624000 chunk: [26624000, 26628096) 26628096, 4096 00020000:00200000:0.0:1789760764.008507:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760764.008520:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6500, 6500] 00000008:00010000:0.0:1789760764.008533:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760764.008537:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760764.008545:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26624000, 26628096) append 0 append lockpos 0 00000080:00200000:0.0:1789760764.008556:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26624000=0x1964000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760764.008659:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760764.008662:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26628096 00000020:00010000:0.0:1789760764.008668:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760764.008671:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.008674:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.008677:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760764.008680:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760764.008685:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760764.008690:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760764.008701:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc780 00000008:00100000:0.0:1789760764.008768:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148700 - object 0x240000400:5 offset 26624000<>26628096 00000100:00100000:1.0:1789760764.009098:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512694144:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760764.076856:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973819 gen 3 00000008:00010000:1.0:1789760764.076894:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760764.076938:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.076949:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.076968:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760764.076985:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512694144:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760764.077081:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6500, 6500] 00000080:00200000:0.0:1789760764.077095:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760764.077100:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760764.077112:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26628096 i_blocks 53255 00000020:00200000:0.0:1789760764.077115:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760764.077123:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760764.077147:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26628096, count: 4096 00000080:00200000:0.0:1789760764.077152:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26628096, bytes: 4096 00000080:00200000:0.0:1789760764.077160:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760764.077167:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760764.077169:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26628096, 26632192) 0 0 00000080:00200000:0.0:1789760764.077172:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760764.077184:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6501, 6501] 00020000:00200000:0.0:1789760764.077190:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26628096 chunk: [26628096, 26632192) 26632192, 4096 00020000:00200000:0.0:1789760764.077194:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760764.077220:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6501, 6501] 00000008:00010000:0.0:1789760764.077234:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760764.077239:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760764.077351:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26628096, 26632192) append 0 append lockpos 0 00000080:00200000:0.0:1789760764.077426:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26628096=0x1965000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760764.077473:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760764.077475:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26632192 00000020:00010000:0.0:1789760764.077483:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760764.077487:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.077489:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.077493:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760764.077495:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760764.077502:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760764.077508:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760764.077522:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000008:00100000:0.0:1789760764.077598:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514aa00 - object 0x240000400:5 offset 26628096<>26632192 00000100:00100000:1.0:1789760764.077783:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512694272:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760764.119622:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973820 gen 3 00000008:00010000:1.0:1789760764.119668:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760764.119675:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.119684:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.119700:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760764.119723:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512694272:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760764.119856:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6501, 6501] 00000080:00200000:2.0:1789760764.119870:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760764.119875:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.119887:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26632192 i_blocks 53255 00000020:00200000:2.0:1789760764.119890:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.119898:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.119957:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26632192, count: 4096 00000080:00200000:2.0:1789760764.119964:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26632192, bytes: 4096 00000080:00200000:2.0:1789760764.119972:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.119979:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.119981:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26632192, 26636288) 0 0 00000080:00200000:2.0:1789760764.119985:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.119996:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6502, 6502] 00020000:00200000:2.0:1789760764.120002:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26632192 chunk: [26632192, 26636288) 26636288, 4096 00020000:00200000:2.0:1789760764.120006:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.120021:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6502, 6502] 00000008:00010000:2.0:1789760764.120034:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.120039:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760764.120055:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26632192, 26636288) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.120065:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26632192=0x1966000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.120109:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.120112:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26636288 00000020:00010000:2.0:1789760764.120118:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.120121:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.120124:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.120127:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.120130:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.120136:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.120142:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.120155:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760764.120228:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 26632192<>26636288 00000100:00100000:2.0:1789760764.120275:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760764.120288:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512694400:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760764.155378:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973821 gen 3 00000008:00010000:3.0:1789760764.155414:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760764.155420:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760764.155428:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760764.155445:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760764.155470:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512694400:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760764.155567:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6502, 6502] 00000080:00200000:2.0:1789760764.155579:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760764.155584:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.155595:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26636288 i_blocks 53255 00000020:00200000:2.0:1789760764.155599:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.155606:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.155631:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26636288, count: 4096 00000080:00200000:2.0:1789760764.155636:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26636288, bytes: 4096 00000080:00200000:2.0:1789760764.155642:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.155647:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.155649:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26636288, 26640384) 0 0 00000080:00200000:2.0:1789760764.155652:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.155661:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6503, 6503] 00020000:00200000:2.0:1789760764.155666:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26636288 chunk: [26636288, 26640384) 26640384, 4096 00020000:00200000:2.0:1789760764.155669:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.155680:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6503, 6503] 00000008:00010000:2.0:1789760764.155741:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.155745:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760764.155753:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26636288, 26640384) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.155762:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26636288=0x1967000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.155849:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.155852:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26640384 00000020:00010000:2.0:1789760764.155859:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.155862:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.155865:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.155869:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.155871:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.155877:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.155883:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.155896:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760764.156158:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 26636288<>26640384 00000100:00100000:3.0:1789760764.156211:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512694528:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760764.205862:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973822 gen 3 00000008:00010000:3.0:1789760764.205898:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760764.205920:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760764.205930:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760764.205955:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760764.205981:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512694528:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760764.206094:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6503, 6503] 00000080:00200000:2.0:1789760764.206107:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760764.206112:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.206124:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26640384 i_blocks 53255 00000020:00200000:2.0:1789760764.206127:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.206136:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.206162:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26640384, count: 4096 00000080:00200000:2.0:1789760764.206168:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26640384, bytes: 4096 00000080:00200000:2.0:1789760764.206179:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.206186:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.206188:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26640384, 26644480) 0 0 00000080:00200000:2.0:1789760764.206192:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.206206:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6504, 6504] 00020000:00200000:2.0:1789760764.206213:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26640384 chunk: [26640384, 26644480) 26644480, 4096 00020000:00200000:2.0:1789760764.206216:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.206232:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6504, 6504] 00000008:00010000:2.0:1789760764.206247:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.206251:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760764.206270:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26640384, 26644480) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.206281:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26640384=0x1968000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.206335:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.206338:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26644480 00000020:00010000:2.0:1789760764.206345:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.206348:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.206351:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.206354:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.206357:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.206375:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.206381:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.206396:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760764.206468:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 26640384<>26644480 00000100:00100000:3.0:1789760764.206674:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512694656:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760764.248670:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760764.248689:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760764.248702:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760764.248734:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000080:00200000:2.0:1789760764.248819:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6504, 6504] 00000080:00200000:2.0:1789760764.248837:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760764.248843:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.248858:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26644480 i_blocks 53255 00000020:00200000:2.0:1789760764.248862:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.248872:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.248936:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26644480, count: 4096 00000080:00200000:2.0:1789760764.248945:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26644480, bytes: 4096 00000080:00200000:2.0:1789760764.248958:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.248968:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.248970:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26644480, 26648576) 0 0 00000080:00200000:2.0:1789760764.248975:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.248993:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6505, 6505] 00020000:00200000:2.0:1789760764.249002:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26644480 chunk: [26644480, 26648576) 26648576, 4096 00020000:00200000:2.0:1789760764.249007:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.249035:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6505, 6505] 00000008:00010000:2.0:1789760764.249056:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.249063:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760764.249076:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26644480, 26648576) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.249092:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26644480=0x1969000, pages 1 (max 131072), lockless, parallel 00000100:00100000:3.0:1789760764.249113:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512694656:192.168.203.156@tcp:4:dd.0 00000020:00200000:2.0:1789760764.249151:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.249154:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26648576 00000008:00010000:3.0:1789760764.249162:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00010000:2.0:1789760764.249164:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.249167:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.249170:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.249174:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.249177:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.249184:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.249191:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00100000:3.0:1789760764.249298:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af480 - object 0x240000400:5 offset 26644480<>26648576 00000100:00100000:3.0:1789760764.249757:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512694784:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760764.298957:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973823 gen 3 00000100:00100000:3.0:1789760764.299005:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc0107af480 x1876699512694784/t4294973824(4294973824) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760780 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760764.299058:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760764.299067:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760764.299078:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760764.299458:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760764.299489:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512694784:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760764.299604:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6505, 6505] 00000080:00200000:2.0:1789760764.299620:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760764.299626:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.299654:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26648576 i_blocks 53255 00000020:00200000:2.0:1789760764.299659:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.299668:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.299703:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26648576, count: 4096 00000080:00200000:2.0:1789760764.299711:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26648576, bytes: 4096 00000080:00200000:2.0:1789760764.299722:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.299731:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.299733:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26648576, 26652672) 0 0 00000080:00200000:2.0:1789760764.299738:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.299755:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6506, 6506] 00020000:00200000:2.0:1789760764.299765:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26648576 chunk: [26648576, 26652672) 26652672, 4096 00020000:00200000:2.0:1789760764.299769:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.299789:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6506, 6506] 00000008:00010000:2.0:1789760764.299806:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.299812:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760764.299823:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26648576, 26652672) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.299837:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26648576=0x196a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.299897:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.301353:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26652672 00000020:00010000:2.0:1789760764.301370:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.301374:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.301378:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.301383:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.301386:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.301401:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.301412:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.301445:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760764.302058:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 26648576<>26652672 00000100:00100000:3.0:1789760764.302526:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512694912:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760764.346893:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973824 gen 3 00000100:00100000:3.0:1789760764.346967:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368dc00 x1876699512694912/t4294973825(4294973825) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760780 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760764.347016:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760764.347024:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760764.347033:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760764.347053:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760764.347081:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512694912:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760764.347457:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6506, 6506] 00000080:00200000:2.0:1789760764.347473:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760764.347479:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.347494:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26652672 i_blocks 53255 00000020:00200000:2.0:1789760764.347497:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.347507:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.347546:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26652672, count: 4096 00000080:00200000:2.0:1789760764.347553:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26652672, bytes: 4096 00000080:00200000:2.0:1789760764.347563:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.347572:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.347575:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26652672, 26656768) 0 0 00000080:00200000:2.0:1789760764.347579:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.347606:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6507, 6507] 00020000:00200000:2.0:1789760764.347614:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26652672 chunk: [26652672, 26656768) 26656768, 4096 00020000:00200000:2.0:1789760764.347619:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.347639:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6507, 6507] 00000008:00010000:2.0:1789760764.347658:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.347664:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760764.347676:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26652672, 26656768) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.347690:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26652672=0x196b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.347749:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.347751:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26656768 00000020:00010000:2.0:1789760764.347759:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.347762:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.347765:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.347769:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.347771:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.347779:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.347786:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.347804:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000008:00100000:2.0:1789760764.348116:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 26652672<>26656768 00000100:00100000:3.0:1789760764.348515:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760764.348540:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512695040:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760764.402415:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973826 gen 3 00000008:00010000:3.0:1789760764.402463:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760764.402470:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760764.402478:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760764.402495:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760764.402516:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512695040:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760764.402589:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6507, 6507] 00000080:00200000:2.0:1789760764.402602:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760764.402606:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.402617:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26656768 i_blocks 53255 00000020:00200000:2.0:1789760764.402621:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.402628:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.402660:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26656768, count: 4096 00000080:00200000:2.0:1789760764.402666:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26656768, bytes: 4096 00000080:00200000:2.0:1789760764.402672:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.402679:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.402681:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26656768, 26660864) 0 0 00000080:00200000:2.0:1789760764.402685:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.402699:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6508, 6508] 00020000:00200000:2.0:1789760764.402705:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26656768 chunk: [26656768, 26660864) 26660864, 4096 00020000:00200000:2.0:1789760764.402708:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.402722:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6508, 6508] 00000008:00010000:2.0:1789760764.402736:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.402740:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760764.402749:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26656768, 26660864) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.402759:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26656768=0x196c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.402798:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.402800:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26660864 00000020:00010000:2.0:1789760764.402807:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.402810:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.402812:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.402815:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.402818:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.402824:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.402828:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.402840:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760764.402954:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 26656768<>26660864 00000100:00100000:3.0:1789760764.403098:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512695168:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760764.435831:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973827 gen 3 00000008:00010000:3.0:1789760764.435869:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760764.435876:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760764.435885:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760764.435932:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760764.435956:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512695168:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760764.436142:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6508, 6508] 00000080:00200000:2.0:1789760764.436155:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760764.436161:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.436172:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26660864 i_blocks 53255 00000020:00200000:2.0:1789760764.436176:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.436184:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.436208:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26660864, count: 4096 00000080:00200000:2.0:1789760764.436215:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26660864, bytes: 4096 00000080:00200000:2.0:1789760764.436222:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.436229:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.436231:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26660864, 26664960) 0 0 00000080:00200000:2.0:1789760764.436235:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.436248:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6509, 6509] 00020000:00200000:2.0:1789760764.436263:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26660864 chunk: [26660864, 26664960) 26664960, 4096 00020000:00200000:2.0:1789760764.436267:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.436281:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6509, 6509] 00000008:00010000:2.0:1789760764.436294:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.436299:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760764.436307:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26660864, 26664960) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.436319:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26660864=0x196d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.436361:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.436364:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26664960 00000020:00010000:2.0:1789760764.436371:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.436374:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.436376:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.436379:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.436382:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.436388:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.436393:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.436408:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760764.436483:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 26660864<>26664960 00000100:00100000:3.0:1789760764.436618:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760764.436631:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512695296:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760764.473521:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760764.473533:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760764.473541:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760764.473561:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760764.473585:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512695296:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760764.473875:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6509, 6509] 00000080:00200000:2.0:1789760764.473887:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760764.473893:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.473940:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26664960 i_blocks 53255 00000020:00200000:2.0:1789760764.473945:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.473953:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.473976:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26664960, count: 4096 00000080:00200000:2.0:1789760764.473983:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26664960, bytes: 4096 00000080:00200000:2.0:1789760764.473990:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.473998:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.474000:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26664960, 26669056) 0 0 00000080:00200000:2.0:1789760764.474004:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.474016:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6510, 6510] 00020000:00200000:2.0:1789760764.474023:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26664960 chunk: [26664960, 26669056) 26669056, 4096 00020000:00200000:2.0:1789760764.474027:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.474041:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6510, 6510] 00000008:00010000:2.0:1789760764.474055:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.474060:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760764.474069:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26664960, 26669056) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.474093:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26664960=0x196e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.474137:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.474140:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26669056 00000020:00010000:2.0:1789760764.474147:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.474150:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.474153:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.474156:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.474159:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.474165:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.474170:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.474184:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000008:00100000:2.0:1789760764.474273:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 26664960<>26669056 00000100:00100000:3.0:1789760764.474526:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512695424:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760764.504887:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973829 gen 3 00000008:00010000:3.0:1789760764.504964:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760764.504972:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760764.504980:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760764.504997:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760764.505020:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512695424:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760764.505280:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6510, 6510] 00000080:00200000:2.0:1789760764.505293:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760764.505297:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.505309:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26669056 i_blocks 53255 00000020:00200000:2.0:1789760764.505313:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.505321:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.505345:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26669056, count: 4096 00000080:00200000:2.0:1789760764.505351:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26669056, bytes: 4096 00000080:00200000:2.0:1789760764.505359:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.505365:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.505367:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26669056, 26673152) 0 0 00000080:00200000:2.0:1789760764.505370:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.505383:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6511, 6511] 00020000:00200000:2.0:1789760764.505390:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26669056 chunk: [26669056, 26673152) 26673152, 4096 00020000:00200000:2.0:1789760764.505394:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.505409:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6511, 6511] 00000008:00010000:2.0:1789760764.505423:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.505427:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760764.505438:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26669056, 26673152) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.505449:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26669056=0x196f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.505493:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.505496:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26673152 00000020:00010000:2.0:1789760764.505502:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.505505:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.505508:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.505511:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.505513:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.505520:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.505525:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.505540:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000008:00100000:2.0:1789760764.505618:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 26669056<>26673152 00000100:00100000:3.0:1789760764.507392:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760764.507416:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512695552:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760764.563638:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973830 gen 3 00000008:00010000:3.0:1789760764.563672:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760764.563678:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760764.563687:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760764.563704:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760764.563727:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512695552:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760764.563930:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6511, 6511] 00000080:00200000:2.0:1789760764.563942:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760764.563946:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.563957:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26673152 i_blocks 53255 00000020:00200000:2.0:1789760764.563960:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.563966:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.563991:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26673152, count: 4096 00000080:00200000:2.0:1789760764.563996:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26673152, bytes: 4096 00000080:00200000:2.0:1789760764.564003:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.564010:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.564012:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26673152, 26677248) 0 0 00000080:00200000:2.0:1789760764.564016:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.564026:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6512, 6512] 00020000:00200000:2.0:1789760764.564032:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26673152 chunk: [26673152, 26677248) 26677248, 4096 00020000:00200000:2.0:1789760764.564036:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.564049:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6512, 6512] 00000008:00010000:2.0:1789760764.564062:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.564066:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760764.564074:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26673152, 26677248) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.564084:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26673152=0x1970000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.564127:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.564129:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26677248 00000020:00010000:2.0:1789760764.564135:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.564138:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.564141:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.564144:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.564146:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.564153:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.564158:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.564171:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760764.564267:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 26673152<>26677248 00000100:00100000:3.0:1789760764.564401:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512695680:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760764.611571:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973831 gen 3 00000008:00010000:3.0:1789760764.611607:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760764.611614:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760764.611622:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760764.611639:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760764.611662:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512695680:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760764.611743:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6512, 6512] 00000080:00200000:2.0:1789760764.611764:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760764.611769:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.611781:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26677248 i_blocks 53255 00000020:00200000:2.0:1789760764.611784:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.611792:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.611817:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26677248, count: 4096 00000080:00200000:2.0:1789760764.613073:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26677248, bytes: 4096 00000080:00200000:2.0:1789760764.613089:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.613097:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.613099:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26677248, 26681344) 0 0 00000080:00200000:2.0:1789760764.613103:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.613116:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6513, 6513] 00020000:00200000:2.0:1789760764.613123:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26677248 chunk: [26677248, 26681344) 26681344, 4096 00020000:00200000:2.0:1789760764.613126:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.613142:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6513, 6513] 00000008:00010000:2.0:1789760764.613157:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.613161:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760764.613170:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26677248, 26681344) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.613180:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26677248=0x1971000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.613228:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.613231:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26681344 00000020:00010000:2.0:1789760764.613238:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.613241:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.613244:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.613665:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.613671:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.613680:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.613686:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.613708:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760764.614092:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 26677248<>26681344 00000100:00100000:3.0:1789760764.614974:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760764.614996:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512695808:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760764.661022:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973832 gen 3 00000008:00010000:3.0:1789760764.661056:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760764.661062:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760764.661070:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760764.661090:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760764.661116:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512695808:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760764.661223:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6513, 6513] 00000080:00200000:2.0:1789760764.661235:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760764.661241:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760764.661261:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26681344 i_blocks 53255 00000020:00200000:2.0:1789760764.661265:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760764.661272:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760764.661296:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26681344, count: 4096 00000080:00200000:2.0:1789760764.661302:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26681344, bytes: 4096 00000080:00200000:2.0:1789760764.661312:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760764.661319:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760764.661321:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26681344, 26685440) 0 0 00000080:00200000:2.0:1789760764.661325:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760764.661337:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6514, 6514] 00020000:00200000:2.0:1789760764.661344:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26681344 chunk: [26681344, 26685440) 26685440, 4096 00020000:00200000:2.0:1789760764.661348:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760764.661362:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6514, 6514] 00000008:00010000:2.0:1789760764.661376:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760764.661381:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760764.661390:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26681344, 26685440) append 0 append lockpos 0 00000080:00200000:2.0:1789760764.661401:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26681344=0x1972000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760764.661454:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760764.661457:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26685440 00000020:00010000:2.0:1789760764.661464:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760764.661466:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.661469:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760764.661472:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760764.661474:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760764.661480:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760764.661485:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760764.661498:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000008:00100000:2.0:1789760764.661571:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 26681344<>26685440 00000100:00100000:3.0:1789760764.661845:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760764.661870:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512695936:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760764.701198:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512696192:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1789760764.701283:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e6cf7267-9b19-462a-a263-14115df15dab:2358:1876699512696064:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1789760764.701328:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2358:1876699512696320:192.168.203.156@tcp:400:kworker.0 00000008:00010000:3.0:1789760764.702338:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760764.702346:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760764.702354:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760764.702384:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760764.702407:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512695936:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760764.702417:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6514, 6514] 00000080:00200000:0.0:1789760764.702431:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760764.702436:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760764.702448:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26685440 i_blocks 53255 00000020:00200000:0.0:1789760764.702452:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760764.702460:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760764.702487:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26685440, count: 4096 00000080:00200000:0.0:1789760764.702494:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26685440, bytes: 4096 00000080:00200000:0.0:1789760764.702502:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760764.702509:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760764.702511:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26685440, 26689536) 0 0 00000080:00200000:0.0:1789760764.702515:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760764.702528:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6515, 6515] 00020000:00200000:0.0:1789760764.702536:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26685440 chunk: [26685440, 26689536) 26689536, 4096 00020000:00200000:0.0:1789760764.702539:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760764.702554:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6515, 6515] 00000008:00010000:0.0:1789760764.702568:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760764.702573:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760764.702582:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26685440, 26689536) append 0 append lockpos 0 00000080:00200000:0.0:1789760764.702594:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26685440=0x1973000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760764.702639:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760764.702642:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26689536 00000020:00010000:0.0:1789760764.702649:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760764.702652:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.702655:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.702658:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760764.702661:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760764.702667:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760764.702673:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760764.702687:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc000 00000008:00100000:0.0:1789760764.702762:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514aa00 - object 0x240000400:5 offset 26685440<>26689536 00010000:00010000:2.0:1789760764.702802:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760764.703048:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512696448:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760764.713575:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e6cf7267-9b19-462a-a263-14115df15dab:2358:1876699512696064:192.168.203.156@tcp:400:kworker.0 00000100:00100000:1.0:1789760764.713608:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512696192:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1789760764.713609:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2358:1876699512696320:192.168.203.156@tcp:400:kworker.0 00000100:00100000:1.0:1789760764.738893:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973834 gen 3 00000008:00010000:1.0:1789760764.738976:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760764.738983:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.738992:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.739001:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760764.739023:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512696448:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760764.740233:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6515, 6515] 00000080:00200000:0.0:1789760764.740269:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760764.740275:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760764.740287:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26689536 i_blocks 53255 00000020:00200000:0.0:1789760764.740291:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760764.740303:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760764.740327:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26689536, count: 4096 00000080:00200000:0.0:1789760764.740333:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26689536, bytes: 4096 00000080:00200000:0.0:1789760764.740343:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760764.740349:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760764.740351:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26689536, 26693632) 0 0 00000080:00200000:0.0:1789760764.740354:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760764.740367:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6516, 6516] 00020000:00200000:0.0:1789760764.740374:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26689536 chunk: [26689536, 26693632) 26693632, 4096 00020000:00200000:0.0:1789760764.740377:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760764.740392:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6516, 6516] 00000008:00010000:0.0:1789760764.740405:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760764.740410:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760764.740418:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26689536, 26693632) append 0 append lockpos 0 00000080:00200000:0.0:1789760764.740428:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26689536=0x1974000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760764.740473:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760764.740475:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26693632 00000020:00010000:0.0:1789760764.740481:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760764.740484:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.740487:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.740490:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760764.740492:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760764.740499:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760764.740504:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760764.740520:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccf00 00000008:00100000:0.0:1789760764.740596:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148700 - object 0x240000400:5 offset 26689536<>26693632 00000100:00100000:1.0:1789760764.742339:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512696576:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760764.832779:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973835 gen 3 00000008:00010000:1.0:1789760764.832815:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760764.832822:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.832832:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.832858:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760764.832883:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512696576:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760764.833030:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6516, 6516] 00000080:00200000:0.0:1789760764.833043:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760764.833048:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760764.833061:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26693632 i_blocks 53255 00000020:00200000:0.0:1789760764.833066:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760764.833074:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760764.833103:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26693632, count: 4096 00000080:00200000:0.0:1789760764.833110:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26693632, bytes: 4096 00000080:00200000:0.0:1789760764.833117:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760764.833124:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760764.833126:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26693632, 26697728) 0 0 00000080:00200000:0.0:1789760764.833130:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760764.833724:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6517, 6517] 00020000:00200000:0.0:1789760764.833734:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26693632 chunk: [26693632, 26697728) 26697728, 4096 00020000:00200000:0.0:1789760764.833738:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760764.833754:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6517, 6517] 00000008:00010000:0.0:1789760764.833768:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760764.833773:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760764.833783:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26693632, 26697728) append 0 append lockpos 0 00000080:00200000:0.0:1789760764.833795:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26693632=0x1975000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760764.833848:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760764.833851:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26697728 00000020:00010000:0.0:1789760764.833857:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760764.833861:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.833864:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.833867:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760764.833870:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760764.833877:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760764.833883:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760764.833930:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cca80 00000008:00100000:0.0:1789760764.834325:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514ad80 - object 0x240000400:5 offset 26693632<>26697728 00000100:00100000:1.0:1789760764.834399:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512696704:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760764.911542:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973836 gen 3 00000008:00010000:1.0:1789760764.911578:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760764.911584:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.911593:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.911670:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760764.911694:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512696704:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760764.912218:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6517, 6517] 00000080:00200000:0.0:1789760764.912232:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760764.912238:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760764.912261:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26697728 i_blocks 53255 00000020:00200000:0.0:1789760764.912265:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760764.912275:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760764.912302:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26697728, count: 4096 00000080:00200000:0.0:1789760764.912309:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26697728, bytes: 4096 00000080:00200000:0.0:1789760764.912317:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760764.912325:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760764.912327:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26697728, 26701824) 0 0 00000080:00200000:0.0:1789760764.912331:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760764.912345:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6518, 6518] 00020000:00200000:0.0:1789760764.912352:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26697728 chunk: [26697728, 26701824) 26701824, 4096 00020000:00200000:0.0:1789760764.912357:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760764.912372:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6518, 6518] 00000008:00010000:0.0:1789760764.912388:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760764.912393:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760764.912402:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26697728, 26701824) append 0 append lockpos 0 00000080:00200000:0.0:1789760764.912414:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26697728=0x1976000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760764.912465:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760764.912468:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26701824 00000020:00010000:0.0:1789760764.912475:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760764.912479:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.912481:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.912484:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760764.912487:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760764.912494:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760764.912499:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760764.912516:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc540 00000008:00100000:0.0:1789760764.912594:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148a80 - object 0x240000400:5 offset 26697728<>26701824 00000100:00100000:1.0:1789760764.912667:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512696832:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760764.989974:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973837 gen 3 00000008:00010000:1.0:1789760764.990011:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760764.990018:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760764.990027:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760764.990049:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760764.990073:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512696832:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760764.990436:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6518, 6518] 00000080:00200000:0.0:1789760764.990448:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760764.990454:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760764.990465:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26701824 i_blocks 53255 00000020:00200000:0.0:1789760764.990468:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760764.990477:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760764.990502:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26701824, count: 4096 00000080:00200000:0.0:1789760764.990508:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26701824, bytes: 4096 00000080:00200000:0.0:1789760764.990515:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760764.990522:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760764.990524:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26701824, 26705920) 0 0 00000080:00200000:0.0:1789760764.990528:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760764.990541:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6519, 6519] 00020000:00200000:0.0:1789760764.990547:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26701824 chunk: [26701824, 26705920) 26705920, 4096 00020000:00200000:0.0:1789760764.990551:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760764.990565:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6519, 6519] 00000008:00010000:0.0:1789760764.990577:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760764.990581:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760764.990588:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26701824, 26705920) append 0 append lockpos 0 00000080:00200000:0.0:1789760764.990599:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26701824=0x1977000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760764.990718:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760764.990721:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26705920 00000020:00010000:0.0:1789760764.990728:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760764.990731:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.990734:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760764.990738:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760764.990740:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760764.990747:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760764.990752:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760764.990766:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc600 00000008:00100000:0.0:1789760764.990840:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148e00 - object 0x240000400:5 offset 26701824<>26705920 00000100:00100000:1.0:1789760764.990915:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760764.990930:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512696960:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760765.038455:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973838 gen 3 00000008:00010000:1.0:1789760765.038487:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760765.038493:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.038502:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.038522:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760765.038544:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512696960:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760765.039297:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6519, 6519] 00000080:00200000:0.0:1789760765.039311:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.039317:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.039329:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26705920 i_blocks 53255 00000020:00200000:0.0:1789760765.039333:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.039341:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.039369:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26705920, count: 4096 00000080:00200000:0.0:1789760765.039375:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26705920, bytes: 4096 00000080:00200000:0.0:1789760765.039384:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.039392:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.039394:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26705920, 26710016) 0 0 00000080:00200000:0.0:1789760765.039398:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.039411:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6520, 6520] 00020000:00200000:0.0:1789760765.039417:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26705920 chunk: [26705920, 26710016) 26710016, 4096 00020000:00200000:0.0:1789760765.039421:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.039436:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6520, 6520] 00000008:00010000:0.0:1789760765.039450:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.039455:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760765.039464:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26705920, 26710016) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.039476:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26705920=0x1978000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.039518:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.039521:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26710016 00000020:00010000:0.0:1789760765.039527:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.039530:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.039533:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.039537:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.039540:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.039546:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.039551:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.039566:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000008:00100000:0.0:1789760765.040345:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a680 - object 0x240000400:5 offset 26705920<>26710016 00000100:00100000:0.0:1789760765.040410:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760765.040429:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512697088:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760765.090948:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973839 gen 3 00000008:00010000:0.0:1789760765.090994:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:0.0:1789760765.091001:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760765.091010:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760765.091032:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:0.0:1789760765.091052:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512697088:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760765.091141:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6520, 6520] 00000080:00200000:2.0:1789760765.091153:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760765.091158:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760765.091172:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26710016 i_blocks 53255 00000020:00200000:2.0:1789760765.091176:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760765.091193:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760765.091220:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26710016, count: 4096 00000080:00200000:2.0:1789760765.091226:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26710016, bytes: 4096 00000080:00200000:2.0:1789760765.091236:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760765.091243:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760765.091254:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26710016, 26714112) 0 0 00000080:00200000:2.0:1789760765.091259:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760765.091273:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6521, 6521] 00020000:00200000:2.0:1789760765.091280:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26710016 chunk: [26710016, 26714112) 26714112, 4096 00020000:00200000:2.0:1789760765.091284:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760765.091298:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6521, 6521] 00000008:00010000:2.0:1789760765.091311:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760765.091315:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760765.091325:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26710016, 26714112) append 0 append lockpos 0 00000080:00200000:2.0:1789760765.091336:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26710016=0x1979000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760765.091376:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760765.091379:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26714112 00000020:00010000:2.0:1789760765.091385:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760765.091388:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760765.091391:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760765.091394:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760765.091396:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760765.091404:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760765.091409:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760765.091431:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760765.091507:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 26710016<>26714112 00000100:00100000:3.0:1789760765.091712:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760765.091728:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512697216:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760765.143608:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760765.143620:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760765.143629:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760765.143687:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760765.143717:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512697216:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760765.148167:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6521, 6521] 00000080:00200000:2.0:1789760765.148188:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760765.148194:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760765.148208:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26714112 i_blocks 53255 00000020:00200000:2.0:1789760765.148214:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760765.148228:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760765.148268:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26714112, count: 4096 00000080:00200000:2.0:1789760765.148614:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26714112, bytes: 4096 00000080:00200000:2.0:1789760765.148628:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760765.148637:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760765.148639:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26714112, 26718208) 0 0 00000080:00200000:2.0:1789760765.148642:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760765.148657:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6522, 6522] 00020000:00200000:2.0:1789760765.148667:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26714112 chunk: [26714112, 26718208) 26718208, 4096 00020000:00200000:2.0:1789760765.148671:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760765.148692:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6522, 6522] 00000008:00010000:2.0:1789760765.148708:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760765.148714:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760765.148727:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26714112, 26718208) append 0 append lockpos 0 00000080:00200000:2.0:1789760765.148739:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26714112=0x197a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760765.148797:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760765.148800:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26718208 00000020:00010000:2.0:1789760765.148807:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760765.148810:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760765.148813:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760765.148816:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760765.148819:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760765.148826:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760765.148832:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760765.148859:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760765.148990:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 26714112<>26718208 00000100:00100000:3.0:1789760765.149076:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512697344:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760765.198094:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973841 gen 3 00000008:00010000:3.0:1789760765.198142:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760765.198150:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760765.198157:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760765.198172:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760765.198195:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512697344:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760765.198309:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6522, 6522] 00000080:00200000:0.0:1789760765.198322:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.198327:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.198339:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26718208 i_blocks 53255 00000020:00200000:0.0:1789760765.198342:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.198350:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.198376:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26718208, count: 4096 00000080:00200000:0.0:1789760765.198382:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26718208, bytes: 4096 00000080:00200000:0.0:1789760765.198392:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.198399:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.198402:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26718208, 26722304) 0 0 00000080:00200000:0.0:1789760765.198406:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.198431:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6523, 6523] 00020000:00200000:0.0:1789760765.198438:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26718208 chunk: [26718208, 26722304) 26722304, 4096 00020000:00200000:0.0:1789760765.198442:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.198460:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6523, 6523] 00000008:00010000:0.0:1789760765.198485:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.198491:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760765.198501:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26718208, 26722304) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.198512:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26718208=0x197b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.198556:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.198559:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26722304 00000020:00010000:0.0:1789760765.198566:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.198569:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.198572:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.198576:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.198578:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.198584:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.198590:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.198605:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000008:00100000:0.0:1789760765.198679:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a680 - object 0x240000400:5 offset 26718208<>26722304 00000100:00100000:1.0:1789760765.198825:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760765.198842:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512697472:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760765.237745:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973842 gen 3 00000008:00010000:1.0:1789760765.237781:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760765.237788:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.237797:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.237818:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760765.237840:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512697472:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760765.238054:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6523, 6523] 00000080:00200000:0.0:1789760765.238066:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.238071:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.238083:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26722304 i_blocks 53255 00000020:00200000:0.0:1789760765.238086:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.238093:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.238120:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26722304, count: 4096 00000080:00200000:0.0:1789760765.238126:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26722304, bytes: 4096 00000080:00200000:0.0:1789760765.238133:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.238141:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.238143:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26722304, 26726400) 0 0 00000080:00200000:0.0:1789760765.238147:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.238160:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6524, 6524] 00020000:00200000:0.0:1789760765.238166:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26722304 chunk: [26722304, 26726400) 26726400, 4096 00020000:00200000:0.0:1789760765.238170:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.238185:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6524, 6524] 00000008:00010000:0.0:1789760765.238200:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.238205:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760765.238214:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26722304, 26726400) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.238225:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26722304=0x197c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.238281:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.238284:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26726400 00000020:00010000:0.0:1789760765.238290:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.238294:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.238297:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.238300:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.238302:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.238309:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.238315:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.238329:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cccc0 00000008:00100000:0.0:1789760765.238402:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a300 - object 0x240000400:5 offset 26722304<>26726400 00000100:00100000:1.0:1789760765.238712:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512697600:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760765.277370:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973843 gen 3 00000008:00010000:1.0:1789760765.277408:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760765.277415:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.277425:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.277446:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760765.277470:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512697600:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760765.277566:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6524, 6524] 00000080:00200000:0.0:1789760765.277578:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.277584:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.277595:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26726400 i_blocks 53255 00000020:00200000:0.0:1789760765.277599:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.277606:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.277631:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26726400, count: 4096 00000080:00200000:0.0:1789760765.277636:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26726400, bytes: 4096 00000080:00200000:0.0:1789760765.277643:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.277650:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.277652:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26726400, 26730496) 0 0 00000080:00200000:0.0:1789760765.277665:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.277685:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6525, 6525] 00020000:00200000:0.0:1789760765.277693:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26726400 chunk: [26726400, 26730496) 26730496, 4096 00020000:00200000:0.0:1789760765.277696:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.277711:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6525, 6525] 00000008:00010000:0.0:1789760765.277725:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.277730:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760765.277738:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26726400, 26730496) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.277748:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26726400=0x197d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.277790:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.277793:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26730496 00000020:00010000:0.0:1789760765.277800:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.277803:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.277806:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.277809:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.277811:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.277817:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.277822:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.277847:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc900 00000008:00100000:0.0:1789760765.277959:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149880 - object 0x240000400:5 offset 26726400<>26730496 00000100:00100000:1.0:1789760765.278243:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760765.278278:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512697728:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760765.328149:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973844 gen 3 00000008:00010000:1.0:1789760765.328186:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760765.328193:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.328203:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.328222:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760765.328257:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512697728:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760765.328346:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6525, 6525] 00000080:00200000:0.0:1789760765.328359:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.328364:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.328376:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26730496 i_blocks 53255 00000020:00200000:0.0:1789760765.328380:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.328388:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.328416:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26730496, count: 4096 00000080:00200000:0.0:1789760765.328423:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26730496, bytes: 4096 00000080:00200000:0.0:1789760765.328431:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.328440:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.328442:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26730496, 26734592) 0 0 00000080:00200000:0.0:1789760765.328446:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.328463:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6526, 6526] 00020000:00200000:0.0:1789760765.328470:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26730496 chunk: [26730496, 26734592) 26734592, 4096 00020000:00200000:0.0:1789760765.328474:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.328490:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6526, 6526] 00000008:00010000:0.0:1789760765.328506:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.328511:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760765.328519:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26730496, 26734592) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.328530:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26730496=0x197e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.328587:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.328590:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26734592 00000020:00010000:0.0:1789760765.328596:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.328600:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.328603:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.328605:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.328608:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.328614:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.328619:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.328633:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000008:00100000:0.0:1789760765.328717:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b480 - object 0x240000400:5 offset 26730496<>26734592 00000100:00100000:1.0:1789760765.329080:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512697856:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760765.391081:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973845 gen 3 00000008:00010000:1.0:1789760765.391118:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760765.391124:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.391134:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.391155:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760765.391177:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512697856:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760765.391298:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6526, 6526] 00000080:00200000:0.0:1789760765.391312:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.391316:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.391327:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26734592 i_blocks 53255 00000020:00200000:0.0:1789760765.391330:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.391338:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.391363:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26734592, count: 4096 00000080:00200000:0.0:1789760765.391369:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26734592, bytes: 4096 00000080:00200000:0.0:1789760765.391376:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.391383:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.391386:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26734592, 26738688) 0 0 00000080:00200000:0.0:1789760765.391390:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.391402:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6527, 6527] 00020000:00200000:0.0:1789760765.391409:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26734592 chunk: [26734592, 26738688) 26738688, 4096 00020000:00200000:0.0:1789760765.391413:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.391428:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6527, 6527] 00000008:00010000:0.0:1789760765.391442:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.391446:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760765.391455:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26734592, 26738688) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.391466:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26734592=0x197f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.391514:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.391516:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26738688 00000020:00010000:0.0:1789760765.391523:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.391525:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.391528:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.391531:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.391534:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.391540:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.391545:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.391558:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccb40 00000008:00100000:0.0:1789760765.391634:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148380 - object 0x240000400:5 offset 26734592<>26738688 00000100:00100000:1.0:1789760765.391720:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760765.391732:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512697984:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760765.438600:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973846 gen 3 00000008:00010000:1.0:1789760765.438636:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760765.438642:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.438651:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.438722:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760765.438745:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512697984:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760765.438817:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6527, 6527] 00000080:00200000:0.0:1789760765.438829:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.438834:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.439398:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26738688 i_blocks 53255 00000020:00200000:0.0:1789760765.439402:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.439410:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.439432:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26738688, count: 4096 00000080:00200000:0.0:1789760765.439438:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26738688, bytes: 4096 00000080:00200000:0.0:1789760765.439445:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.439451:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.439453:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26738688, 26742784) 0 0 00000080:00200000:0.0:1789760765.439457:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.439468:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6528, 6528] 00020000:00200000:0.0:1789760765.439475:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26738688 chunk: [26738688, 26742784) 26742784, 4096 00020000:00200000:0.0:1789760765.439478:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.439490:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6528, 6528] 00000008:00010000:0.0:1789760765.439504:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.439508:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760765.439516:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26738688, 26742784) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.439526:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26738688=0x1980000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.439569:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.439571:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26742784 00000020:00010000:0.0:1789760765.439578:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.439581:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.439585:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.439587:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.439590:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.439596:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.439602:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.439615:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc300 00000008:00100000:0.0:1789760765.439689:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b800 - object 0x240000400:5 offset 26738688<>26742784 00000100:00100000:1.0:1789760765.439754:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512698112:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760765.510144:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973847 gen 3 00000008:00010000:1.0:1789760765.510267:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760765.510275:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.510287:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.510545:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760765.510572:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512698112:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760765.511570:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6528, 6528] 00000080:00200000:0.0:1789760765.511584:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760765.511589:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760765.511602:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26742784 i_blocks 53255 00000020:00200000:0.0:1789760765.511606:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760765.511616:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760765.511641:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26742784, count: 4096 00000080:00200000:0.0:1789760765.511647:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26742784, bytes: 4096 00000080:00200000:0.0:1789760765.511658:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760765.511665:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760765.511667:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26742784, 26746880) 0 0 00000080:00200000:0.0:1789760765.511671:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760765.511685:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6529, 6529] 00020000:00200000:0.0:1789760765.511691:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26742784 chunk: [26742784, 26746880) 26746880, 4096 00020000:00200000:0.0:1789760765.511695:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760765.511710:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6529, 6529] 00000008:00010000:0.0:1789760765.511723:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760765.511729:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760765.511738:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26742784, 26746880) append 0 append lockpos 0 00000080:00200000:0.0:1789760765.511748:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26742784=0x1981000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760765.511792:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760765.511795:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26746880 00000020:00010000:0.0:1789760765.511802:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760765.511805:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.511808:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760765.511811:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760765.511814:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760765.511822:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760765.511828:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760765.511845:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc240 00000008:00100000:0.0:1789760765.515384:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149c00 - object 0x240000400:5 offset 26742784<>26746880 00000100:00100000:0.0:1789760765.515465:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512698240:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760765.561202:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973848 gen 3 00000008:00010000:0.0:1789760765.561240:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:0.0:1789760765.561257:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760765.561267:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760765.561289:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:0.0:1789760765.561311:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512698240:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760765.561422:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6529, 6529] 00000080:00200000:1.0:1789760765.561434:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760765.561439:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760765.561450:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26746880 i_blocks 53255 00000020:00200000:1.0:1789760765.561454:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760765.561475:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760765.561500:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26746880, count: 4096 00000080:00200000:1.0:1789760765.561507:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26746880, bytes: 4096 00000080:00200000:1.0:1789760765.561514:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760765.561520:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760765.561522:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26746880, 26750976) 0 0 00000080:00200000:1.0:1789760765.561525:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760765.561540:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6530, 6530] 00020000:00200000:1.0:1789760765.561546:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26746880 chunk: [26746880, 26750976) 26750976, 4096 00020000:00200000:1.0:1789760765.561550:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760765.561564:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6530, 6530] 00000008:00010000:1.0:1789760765.561576:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760765.561581:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760765.561589:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26746880, 26750976) append 0 append lockpos 0 00000080:00200000:1.0:1789760765.561599:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26746880=0x1982000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760765.561658:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760765.561661:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26750976 00000020:00010000:1.0:1789760765.561667:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760765.561670:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760765.561673:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760765.561677:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760765.561680:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760765.561685:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760765.561690:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760765.561704:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760765.561792:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96680 - object 0x240000400:5 offset 26746880<>26750976 00000100:00100000:0.0:1789760765.562042:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512698368:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760765.627691:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973849 gen 3 00000008:00010000:0.0:1789760765.627733:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760765.627740:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760765.627749:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760765.627773:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760765.627798:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512698368:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760765.629537:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6530, 6530] 00000080:00200000:1.0:1789760765.629554:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760765.629560:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760765.629574:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26750976 i_blocks 53255 00000020:00200000:1.0:1789760765.629578:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760765.629589:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760765.629613:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26750976, count: 4096 00000080:00200000:1.0:1789760765.629619:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26750976, bytes: 4096 00000080:00200000:1.0:1789760765.629630:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760765.629638:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760765.629639:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26750976, 26755072) 0 0 00000080:00200000:1.0:1789760765.629643:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760765.629656:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6531, 6531] 00020000:00200000:1.0:1789760765.629664:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26750976 chunk: [26750976, 26755072) 26755072, 4096 00020000:00200000:1.0:1789760765.629668:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760765.629684:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6531, 6531] 00000008:00010000:1.0:1789760765.629697:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760765.629703:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760765.629711:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26750976, 26755072) append 0 append lockpos 0 00000080:00200000:1.0:1789760765.629722:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26750976=0x1983000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760765.629771:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760765.629773:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26755072 00000020:00010000:1.0:1789760765.629780:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760765.629783:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760765.629786:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760765.629788:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760765.629791:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760765.629797:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760765.629804:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760765.629833:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760765.629954:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95880 - object 0x240000400:5 offset 26750976<>26755072 00000100:00100000:1.0:1789760765.630000:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760765.630019:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512698496:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760765.705465:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973850 gen 3 00000008:00010000:1.0:1789760765.705499:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760765.705507:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760765.705517:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760765.705540:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760765.705563:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512698496:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760765.705652:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6531, 6531] 00000080:00200000:2.0:1789760765.705664:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760765.705670:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760765.705682:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26755072 i_blocks 53255 00000020:00200000:2.0:1789760765.705686:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760765.705693:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760765.705721:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26755072, count: 4096 00000080:00200000:2.0:1789760765.705727:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26755072, bytes: 4096 00000080:00200000:2.0:1789760765.705735:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760765.705742:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760765.705744:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26755072, 26759168) 0 0 00000080:00200000:2.0:1789760765.705748:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760765.705761:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6532, 6532] 00020000:00200000:2.0:1789760765.705767:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26755072 chunk: [26755072, 26759168) 26759168, 4096 00020000:00200000:2.0:1789760765.705771:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760765.705785:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6532, 6532] 00000008:00010000:2.0:1789760765.705813:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760765.705818:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760765.705827:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26755072, 26759168) append 0 append lockpos 0 00000080:00200000:2.0:1789760765.705837:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26755072=0x1984000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760765.705880:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760765.705883:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26759168 00000020:00010000:2.0:1789760765.705890:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760765.705893:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760765.705896:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760765.705934:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760765.705938:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760765.705944:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760765.705950:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760765.705964:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760765.706453:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 26755072<>26759168 00000100:00100000:2.0:1789760765.706499:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512698624:192.168.203.156@tcp:4:dd.0 00010000:00010000:0.0:1789760765.725111:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:3.0:1789760765.742491:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760765.742506:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760765.742515:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760765.742536:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760765.742564:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512698624:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760765.742963:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6532, 6532] 00000080:00200000:1.0:1789760765.742978:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760765.742984:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760765.742998:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26759168 i_blocks 53255 00000020:00200000:1.0:1789760765.743002:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760765.743011:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760765.743039:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26759168, count: 4096 00000080:00200000:1.0:1789760765.743045:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26759168, bytes: 4096 00000080:00200000:1.0:1789760765.743054:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760765.743063:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760765.743064:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26759168, 26763264) 0 0 00000080:00200000:1.0:1789760765.743068:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760765.743082:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6533, 6533] 00020000:00200000:1.0:1789760765.743090:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26759168 chunk: [26759168, 26763264) 26763264, 4096 00020000:00200000:1.0:1789760765.743093:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760765.743110:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6533, 6533] 00000008:00010000:1.0:1789760765.743122:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760765.743126:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760765.743135:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26759168, 26763264) append 0 append lockpos 0 00000080:00200000:1.0:1789760765.743146:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26759168=0x1985000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760765.743197:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760765.743200:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26763264 00000020:00010000:1.0:1789760765.743207:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760765.743211:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760765.743214:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760765.743217:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760765.743221:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760765.743228:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760765.743234:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760765.743257:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760765.743343:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95880 - object 0x240000400:5 offset 26759168<>26763264 00000100:00100000:0.0:1789760765.743454:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512698752:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760765.788977:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973852 gen 3 00000008:00010000:0.0:1789760765.789023:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760765.789030:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760765.789038:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760765.789058:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760765.789081:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512698752:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760765.789156:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6533, 6533] 00000080:00200000:3.0:1789760765.789169:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760765.789173:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760765.789185:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26763264 i_blocks 53255 00000020:00200000:3.0:1789760765.789188:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760765.789195:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760765.789219:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26763264, count: 4096 00000080:00200000:3.0:1789760765.789225:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26763264, bytes: 4096 00000080:00200000:3.0:1789760765.789232:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760765.789239:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760765.789249:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26763264, 26767360) 0 0 00000080:00200000:3.0:1789760765.789253:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760765.789264:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6534, 6534] 00020000:00200000:3.0:1789760765.789271:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26763264 chunk: [26763264, 26767360) 26767360, 4096 00020000:00200000:3.0:1789760765.789275:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760765.789289:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6534, 6534] 00000008:00010000:3.0:1789760765.789304:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760765.789308:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760765.789317:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26763264, 26767360) append 0 append lockpos 0 00000080:00200000:3.0:1789760765.789327:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26763264=0x1986000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760765.789367:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760765.789369:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26767360 00000020:00010000:3.0:1789760765.789375:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760765.789378:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.789381:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.789384:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760765.789387:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760765.789393:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760765.789398:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760765.789411:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760765.789483:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af480 - object 0x240000400:5 offset 26763264<>26767360 00000100:00100000:3.0:1789760765.789525:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760765.789536:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512698880:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760765.842821:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760765.842831:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760765.842840:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760765.842858:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760765.842883:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512698880:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760765.843035:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6534, 6534] 00000080:00200000:3.0:1789760765.843049:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760765.843055:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760765.843066:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26767360 i_blocks 53255 00000020:00200000:3.0:1789760765.843069:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760765.843076:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760765.843101:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26767360, count: 4096 00000080:00200000:3.0:1789760765.843108:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26767360, bytes: 4096 00000080:00200000:3.0:1789760765.843115:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760765.843122:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760765.843125:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26767360, 26771456) 0 0 00000080:00200000:3.0:1789760765.843129:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760765.843141:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6535, 6535] 00020000:00200000:3.0:1789760765.843147:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26767360 chunk: [26767360, 26771456) 26771456, 4096 00020000:00200000:3.0:1789760765.843151:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760765.843164:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6535, 6535] 00000008:00010000:3.0:1789760765.843178:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760765.843183:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760765.843192:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26767360, 26771456) append 0 append lockpos 0 00000080:00200000:3.0:1789760765.843202:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26767360=0x1987000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760765.843236:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760765.843239:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26771456 00000020:00010000:3.0:1789760765.843258:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760765.843262:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.843265:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.843268:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760765.843271:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760765.843277:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760765.843282:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760765.843355:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:2.0:1789760765.843430:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 26767360<>26771456 00000100:00100000:3.0:1789760765.843479:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760765.844103:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512699008:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760765.892516:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973854 gen 3 00000008:00010000:2.0:1789760765.892564:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760765.892571:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760765.892580:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760765.892607:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760765.892628:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512699008:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760765.892832:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6535, 6535] 00000080:00200000:3.0:1789760765.892845:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760765.892850:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760765.892861:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26771456 i_blocks 53255 00000020:00200000:3.0:1789760765.892865:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760765.892872:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760765.892897:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26771456, count: 4096 00000080:00200000:3.0:1789760765.892925:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26771456, bytes: 4096 00000080:00200000:3.0:1789760765.892935:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760765.892942:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760765.892945:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26771456, 26775552) 0 0 00000080:00200000:3.0:1789760765.892948:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760765.892961:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6536, 6536] 00020000:00200000:3.0:1789760765.892967:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26771456 chunk: [26771456, 26775552) 26775552, 4096 00020000:00200000:3.0:1789760765.892971:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760765.892987:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6536, 6536] 00000008:00010000:3.0:1789760765.893001:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760765.893006:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760765.893016:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26771456, 26775552) append 0 append lockpos 0 00000080:00200000:3.0:1789760765.893028:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26771456=0x1988000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760765.893073:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760765.893075:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26775552 00000020:00010000:3.0:1789760765.893082:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760765.893085:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.893088:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.893091:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760765.893093:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760765.893099:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760765.893105:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760765.893118:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760765.893192:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae300 - object 0x240000400:5 offset 26771456<>26775552 00000100:00100000:2.0:1789760765.893368:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512699136:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760765.932972:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973855 gen 3 00000008:00010000:2.0:1789760765.933012:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760765.933019:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760765.933028:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760765.933046:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760765.933138:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512699136:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760765.933777:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6536, 6536] 00000080:00200000:3.0:1789760765.933791:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760765.933795:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760765.933807:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26775552 i_blocks 53255 00000020:00200000:3.0:1789760765.933810:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760765.933818:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760765.933843:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26775552, count: 4096 00000080:00200000:3.0:1789760765.933849:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26775552, bytes: 4096 00000080:00200000:3.0:1789760765.933857:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760765.933863:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760765.933866:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26775552, 26779648) 0 0 00000080:00200000:3.0:1789760765.933869:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760765.933881:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6537, 6537] 00020000:00200000:3.0:1789760765.933888:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26775552 chunk: [26775552, 26779648) 26779648, 4096 00020000:00200000:3.0:1789760765.933891:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760765.933935:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6537, 6537] 00000008:00010000:3.0:1789760765.933952:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760765.933958:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760765.933967:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26775552, 26779648) append 0 append lockpos 0 00000080:00200000:3.0:1789760765.933978:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26775552=0x1989000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760765.934023:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760765.934026:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26779648 00000020:00010000:3.0:1789760765.934033:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760765.934036:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.934038:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760765.934041:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760765.934044:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760765.934051:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760765.934056:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760765.935637:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760765.935869:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac700 - object 0x240000400:5 offset 26775552<>26779648 00000100:00100000:3.0:1789760765.935949:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760765.935971:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512699264:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760765.976634:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973856 gen 3 00000008:00010000:2.0:1789760765.976672:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760765.976679:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760765.976689:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760765.976713:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760765.976738:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512699264:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760765.976819:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6537, 6537] 00000080:00200000:1.0:1789760765.976832:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760765.976836:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760765.976848:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26779648 i_blocks 53255 00000020:00200000:1.0:1789760765.976851:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760765.976859:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760765.976890:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26779648, count: 4096 00000080:00200000:1.0:1789760765.976896:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26779648, bytes: 4096 00000080:00200000:1.0:1789760765.976925:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760765.976933:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760765.976935:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26779648, 26783744) 0 0 00000080:00200000:1.0:1789760765.976939:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760765.976952:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6538, 6538] 00020000:00200000:1.0:1789760765.976959:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26779648 chunk: [26779648, 26783744) 26783744, 4096 00020000:00200000:1.0:1789760765.976963:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760765.976979:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6538, 6538] 00000008:00010000:1.0:1789760765.976993:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760765.976997:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760765.977005:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26779648, 26783744) append 0 append lockpos 0 00000080:00200000:1.0:1789760765.977015:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26779648=0x198a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760765.977069:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760765.977071:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26783744 00000020:00010000:1.0:1789760765.977078:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760765.977082:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760765.977085:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760765.977088:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760765.977091:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760765.977097:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760765.977102:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760765.977115:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760765.977196:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96d80 - object 0x240000400:5 offset 26779648<>26783744 00000100:00100000:0.0:1789760765.977284:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512699392:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760766.010149:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973857 gen 3 00000008:00010000:0.0:1789760766.010179:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760766.010186:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760766.010193:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760766.010214:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760766.010236:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512699392:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760766.010436:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6538, 6538] 00000080:00200000:1.0:1789760766.010449:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760766.010454:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760766.010465:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26783744 i_blocks 53255 00000020:00200000:1.0:1789760766.010468:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760766.010477:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760766.010502:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26783744, count: 4096 00000080:00200000:1.0:1789760766.010508:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26783744, bytes: 4096 00000080:00200000:1.0:1789760766.010516:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760766.010524:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760766.010526:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26783744, 26787840) 0 0 00000080:00200000:1.0:1789760766.010529:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760766.010542:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6539, 6539] 00020000:00200000:1.0:1789760766.010548:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26783744 chunk: [26783744, 26787840) 26787840, 4096 00020000:00200000:1.0:1789760766.010552:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760766.010566:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6539, 6539] 00000008:00010000:1.0:1789760766.010578:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760766.010582:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760766.010590:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26783744, 26787840) append 0 append lockpos 0 00000080:00200000:1.0:1789760766.010601:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26783744=0x198b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760766.010700:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760766.010703:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26787840 00000020:00010000:1.0:1789760766.010709:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760766.010713:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.010716:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.010719:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760766.010721:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760766.010727:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760766.010732:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760766.010744:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760766.010875:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96d80 - object 0x240000400:5 offset 26783744<>26787840 00000100:00100000:0.0:1789760766.011015:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512699520:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760766.036146:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973858 gen 3 00000008:00010000:0.0:1789760766.036182:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760766.036189:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760766.036208:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760766.036235:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760766.036269:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512699520:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760766.036287:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6539, 6539] 00000080:00200000:2.0:1789760766.036375:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.036381:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.036394:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26787840 i_blocks 53255 00000020:00200000:2.0:1789760766.036398:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.036408:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.036434:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26787840, count: 4096 00000080:00200000:2.0:1789760766.036441:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26787840, bytes: 4096 00000080:00200000:2.0:1789760766.036449:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.036456:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.036458:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26787840, 26791936) 0 0 00000080:00200000:2.0:1789760766.036462:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.036474:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6540, 6540] 00020000:00200000:2.0:1789760766.036481:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26787840 chunk: [26787840, 26791936) 26791936, 4096 00020000:00200000:2.0:1789760766.036485:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.036499:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6540, 6540] 00000008:00010000:2.0:1789760766.036515:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.036520:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760766.036529:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26787840, 26791936) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.036542:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26787840=0x198c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.036589:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.036592:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26791936 00000020:00010000:2.0:1789760766.036599:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.036602:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.036605:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.036609:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.036611:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.036618:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.036624:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760766.036638:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760766.036715:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 26787840<>26791936 00000100:00100000:2.0:1789760766.036752:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760766.036766:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512699648:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760766.077342:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973859 gen 3 00000008:00010000:2.0:1789760766.077380:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:2.0:1789760766.077387:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760766.077396:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760766.077422:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:2.0:1789760766.077448:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512699648:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760766.078114:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6540, 6540] 00000080:00200000:0.0:1789760766.078129:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.078134:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.078146:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26791936 i_blocks 53255 00000020:00200000:0.0:1789760766.078150:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.078159:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.078185:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26791936, count: 4096 00000080:00200000:0.0:1789760766.078191:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26791936, bytes: 4096 00000080:00200000:0.0:1789760766.078200:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.078208:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.078211:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26791936, 26796032) 0 0 00000080:00200000:0.0:1789760766.078215:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.078228:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6541, 6541] 00020000:00200000:0.0:1789760766.078235:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26791936 chunk: [26791936, 26796032) 26796032, 4096 00020000:00200000:0.0:1789760766.078251:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.078266:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6541, 6541] 00000008:00010000:0.0:1789760766.078279:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.078284:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760766.078293:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26791936, 26796032) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.078314:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26791936=0x198d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.078356:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.078358:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26796032 00000020:00010000:0.0:1789760766.078364:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.078367:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.078369:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.078372:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.078374:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.078381:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.078386:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760766.078402:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc240 00000008:00100000:0.0:1789760766.078474:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149c00 - object 0x240000400:5 offset 26791936<>26796032 00000100:00100000:0.0:1789760766.078518:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760766.078532:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512699776:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760766.120730:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:0.0:1789760766.120741:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760766.120750:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760766.120774:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000080:00200000:2.0:1789760766.120939:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6541, 6541] 00000080:00200000:2.0:1789760766.120955:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.120959:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.120982:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26796032 i_blocks 53255 00000020:00200000:2.0:1789760766.120985:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.120993:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.121018:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26796032, count: 4096 00000080:00200000:2.0:1789760766.121025:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26796032, bytes: 4096 00000080:00200000:2.0:1789760766.121033:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.121040:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.121045:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26796032, 26800128) 0 0 00000080:00200000:2.0:1789760766.121050:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.121064:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6542, 6542] 00020000:00200000:2.0:1789760766.121070:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26796032 chunk: [26796032, 26800128) 26800128, 4096 00020000:00200000:2.0:1789760766.121074:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.121088:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6542, 6542] 00000008:00010000:2.0:1789760766.121102:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.121106:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760766.121115:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26796032, 26800128) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.121125:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26796032=0x198e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.121160:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.121163:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26800128 00000020:00010000:2.0:1789760766.121169:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.121172:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.121175:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.121178:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.121181:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.121187:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.121192:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1789760766.121407:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512699776:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760766.121437:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:0.0:1789760766.121518:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149180 - object 0x240000400:5 offset 26796032<>26800128 00000100:00100000:0.0:1789760766.121585:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512699904:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760766.173312:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973861 gen 3 00000008:00010000:0.0:1789760766.173375:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:0.0:1789760766.173382:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760766.173390:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760766.173409:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:0.0:1789760766.173429:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512699904:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760766.173675:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6542, 6542] 00000080:00200000:2.0:1789760766.173687:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.173692:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.173704:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26800128 i_blocks 53255 00000020:00200000:2.0:1789760766.173707:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.173716:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.173741:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26800128, count: 4096 00000080:00200000:2.0:1789760766.173748:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26800128, bytes: 4096 00000080:00200000:2.0:1789760766.173756:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.173763:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.173765:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26800128, 26804224) 0 0 00000080:00200000:2.0:1789760766.173769:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.173781:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6543, 6543] 00020000:00200000:2.0:1789760766.173851:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26800128 chunk: [26800128, 26804224) 26804224, 4096 00020000:00200000:2.0:1789760766.173856:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.173871:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6543, 6543] 00000008:00010000:2.0:1789760766.173885:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.173890:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760766.173931:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26800128, 26804224) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.173943:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26800128=0x198f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.173985:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.173988:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26804224 00000020:00010000:2.0:1789760766.173995:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.173998:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.174001:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.174004:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.174007:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.174013:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.174018:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760766.174033:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760766.174105:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 26800128<>26804224 00000100:00100000:3.0:1789760766.174175:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512700032:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760766.206090:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760766.206103:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760766.206112:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760766.206136:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760766.206160:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512700032:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760766.206445:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6543, 6543] 00000080:00200000:0.0:1789760766.206458:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760766.206463:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760766.206475:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26804224 i_blocks 53255 00000020:00200000:0.0:1789760766.206478:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760766.206485:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760766.206511:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26804224, count: 4096 00000080:00200000:0.0:1789760766.206518:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26804224, bytes: 4096 00000080:00200000:0.0:1789760766.206526:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760766.206533:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760766.206535:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26804224, 26808320) 0 0 00000080:00200000:0.0:1789760766.206539:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760766.206552:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6544, 6544] 00020000:00200000:0.0:1789760766.206561:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26804224 chunk: [26804224, 26808320) 26808320, 4096 00020000:00200000:0.0:1789760766.206565:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760766.206580:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6544, 6544] 00000008:00010000:0.0:1789760766.206594:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760766.206599:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760766.206608:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26804224, 26808320) append 0 append lockpos 0 00000080:00200000:0.0:1789760766.206619:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26804224=0x1990000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760766.206723:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760766.206726:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26808320 00000020:00010000:0.0:1789760766.206733:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760766.206736:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.206739:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760766.206742:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760766.206745:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760766.206751:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760766.206756:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760766.206769:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc240 00000008:00100000:0.0:1789760766.206860:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149180 - object 0x240000400:5 offset 26804224<>26808320 00000100:00100000:0.0:1789760766.207017:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760766.207032:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512700160:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760766.276968:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973863 gen 3 00000008:00010000:0.0:1789760766.277021:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:0.0:1789760766.277028:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760766.277036:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760766.277059:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:0.0:1789760766.277080:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512700160:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760766.277295:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6544, 6544] 00000080:00200000:1.0:1789760766.277309:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760766.277314:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760766.277327:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26808320 i_blocks 53255 00000020:00200000:1.0:1789760766.277330:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760766.277338:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760766.277364:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26808320, count: 4096 00000080:00200000:1.0:1789760766.277370:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26808320, bytes: 4096 00000080:00200000:1.0:1789760766.277378:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760766.277386:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760766.277389:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26808320, 26812416) 0 0 00000080:00200000:1.0:1789760766.277392:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760766.277404:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6545, 6545] 00020000:00200000:1.0:1789760766.277411:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26808320 chunk: [26808320, 26812416) 26812416, 4096 00020000:00200000:1.0:1789760766.277414:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760766.277428:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6545, 6545] 00000008:00010000:1.0:1789760766.277440:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760766.277445:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760766.277453:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26808320, 26812416) append 0 append lockpos 0 00000080:00200000:1.0:1789760766.277463:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26808320=0x1991000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760766.277508:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760766.277511:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26812416 00000020:00010000:1.0:1789760766.277518:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760766.277521:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.277524:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.277527:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760766.277530:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760766.277536:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760766.277542:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760766.277557:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760766.277635:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95880 - object 0x240000400:5 offset 26808320<>26812416 00000100:00100000:0.0:1789760766.277700:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512700288:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760766.324707:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973864 gen 3 00000008:00010000:0.0:1789760766.324744:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760766.324750:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760766.324758:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760766.324832:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760766.324858:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512700288:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760766.327047:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6545, 6545] 00000080:00200000:1.0:1789760766.327066:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760766.327071:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760766.327085:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26812416 i_blocks 53255 00000020:00200000:1.0:1789760766.327089:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760766.327105:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760766.327134:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26812416, count: 4096 00000080:00200000:1.0:1789760766.327140:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26812416, bytes: 4096 00000080:00200000:1.0:1789760766.327163:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760766.327172:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760766.327174:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26812416, 26816512) 0 0 00000080:00200000:1.0:1789760766.327177:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760766.327191:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6546, 6546] 00020000:00200000:1.0:1789760766.327206:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26812416 chunk: [26812416, 26816512) 26816512, 4096 00020000:00200000:1.0:1789760766.327210:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760766.327512:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6546, 6546] 00000008:00010000:1.0:1789760766.327529:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760766.327535:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760766.327544:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26812416, 26816512) append 0 append lockpos 0 00000080:00200000:1.0:1789760766.327557:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26812416=0x1992000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760766.327611:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760766.327614:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26816512 00000020:00010000:1.0:1789760766.327620:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760766.327623:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.327626:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.327629:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760766.327631:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760766.327637:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760766.327643:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760766.327667:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760766.327771:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96680 - object 0x240000400:5 offset 26812416<>26816512 00000100:00100000:0.0:1789760766.328054:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760766.328075:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512700416:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760766.392395:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973865 gen 3 00000008:00010000:0.0:1789760766.392435:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760766.392442:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760766.392450:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760766.392476:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760766.392501:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512700416:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760766.393081:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6546, 6546] 00000080:00200000:1.0:1789760766.393094:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760766.393099:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760766.393109:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26816512 i_blocks 53255 00000020:00200000:1.0:1789760766.393112:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760766.393121:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760766.393147:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26816512, count: 4096 00000080:00200000:1.0:1789760766.393153:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26816512, bytes: 4096 00000080:00200000:1.0:1789760766.393165:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760766.393173:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760766.393175:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26816512, 26820608) 0 0 00000080:00200000:1.0:1789760766.393179:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760766.393191:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6547, 6547] 00020000:00200000:1.0:1789760766.393198:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26816512 chunk: [26816512, 26820608) 26820608, 4096 00020000:00200000:1.0:1789760766.393202:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760766.393217:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6547, 6547] 00000008:00010000:1.0:1789760766.393230:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760766.393235:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760766.393255:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26816512, 26820608) append 0 append lockpos 0 00000080:00200000:1.0:1789760766.393266:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26816512=0x1993000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760766.393310:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760766.393313:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26820608 00000020:00010000:1.0:1789760766.393319:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760766.393322:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.393325:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.393328:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760766.393330:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760766.393337:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760766.393342:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760766.393356:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760766.393431:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97480 - object 0x240000400:5 offset 26816512<>26820608 00000100:00100000:0.0:1789760766.393524:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760766.393542:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512700544:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760766.468674:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760766.468697:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760766.468706:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760766.468924:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760766.468955:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512700544:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760766.470744:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6547, 6547] 00000080:00200000:1.0:1789760766.470762:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760766.470767:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760766.470782:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26820608 i_blocks 53255 00000020:00200000:1.0:1789760766.470786:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760766.470798:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760766.470827:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26820608, count: 4096 00000080:00200000:1.0:1789760766.470835:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26820608, bytes: 4096 00000080:00200000:1.0:1789760766.470847:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760766.470855:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760766.470857:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26820608, 26824704) 0 0 00000080:00200000:1.0:1789760766.470861:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760766.470873:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6548, 6548] 00020000:00200000:1.0:1789760766.470879:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26820608 chunk: [26820608, 26824704) 26824704, 4096 00020000:00200000:1.0:1789760766.470883:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760766.470898:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6548, 6548] 00000008:00010000:1.0:1789760766.470939:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760766.470944:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760766.470952:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26820608, 26824704) append 0 append lockpos 0 00000080:00200000:1.0:1789760766.470963:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26820608=0x1994000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760766.471018:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760766.471021:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26824704 00000020:00010000:1.0:1789760766.471028:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760766.471031:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.471034:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.471037:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760766.471039:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760766.471046:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760766.471051:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760766.471068:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760766.471153:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 26820608<>26824704 00000100:00100000:0.0:1789760766.471524:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760766.471543:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512700672:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760766.503209:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973866 gen 3 00000100:00100000:0.0:1789760766.503250:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005c94e00 x1876699512700672/t4294973867(4294973867) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760782 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760766.503289:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760766.503297:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760766.503304:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760766.503336:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760766.503356:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512700672:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760766.503534:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6548, 6548] 00000080:00200000:1.0:1789760766.503545:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760766.503550:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760766.503562:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26824704 i_blocks 53255 00000020:00200000:1.0:1789760766.503565:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760766.503573:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760766.503601:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26824704, count: 4096 00000080:00200000:1.0:1789760766.503608:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26824704, bytes: 4096 00000080:00200000:1.0:1789760766.503618:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760766.503626:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760766.503628:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26824704, 26828800) 0 0 00000080:00200000:1.0:1789760766.503632:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760766.503644:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6549, 6549] 00020000:00200000:1.0:1789760766.503652:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26824704 chunk: [26824704, 26828800) 26828800, 4096 00020000:00200000:1.0:1789760766.503656:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760766.503673:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6549, 6549] 00000008:00010000:1.0:1789760766.503687:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760766.503691:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760766.503700:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26824704, 26828800) append 0 append lockpos 0 00000080:00200000:1.0:1789760766.503710:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26824704=0x1995000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760766.503755:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760766.503758:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26828800 00000020:00010000:1.0:1789760766.503765:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760766.503768:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.503771:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.503774:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760766.503777:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760766.503783:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760766.503788:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760766.503801:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760766.503882:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 26824704<>26828800 00000100:00100000:0.0:1789760766.504247:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760766.504262:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512700800:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760766.546131:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973867 gen 3 00000100:00100000:0.0:1789760766.546174:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005c97b80 x1876699512700800/t4294973868(4294973868) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760782 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760766.546212:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760766.546218:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760766.546225:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760766.546250:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760766.546274:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512700800:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760766.546485:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6549, 6549] 00000080:00200000:1.0:1789760766.546497:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760766.546501:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760766.546513:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26828800 i_blocks 53255 00000020:00200000:1.0:1789760766.546516:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760766.546525:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760766.546552:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26828800, count: 4096 00000080:00200000:1.0:1789760766.546558:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26828800, bytes: 4096 00000080:00200000:1.0:1789760766.546567:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760766.546574:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760766.546576:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26828800, 26832896) 0 0 00000080:00200000:1.0:1789760766.546581:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760766.546595:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6550, 6550] 00020000:00200000:1.0:1789760766.546601:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26828800 chunk: [26828800, 26832896) 26832896, 4096 00020000:00200000:1.0:1789760766.546605:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760766.546631:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6550, 6550] 00000008:00010000:1.0:1789760766.546646:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760766.546650:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760766.546659:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26828800, 26832896) append 0 append lockpos 0 00000080:00200000:1.0:1789760766.546672:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26828800=0x1996000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760766.546717:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760766.546720:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26832896 00000020:00010000:1.0:1789760766.546726:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760766.546729:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.546731:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.546734:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760766.546736:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760766.546743:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760766.546748:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760766.546762:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760766.546844:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 26828800<>26832896 00000100:00100000:0.0:1789760766.546996:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760766.547014:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512700928:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760766.611495:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973868 gen 3 00000100:00100000:0.0:1789760766.611525:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005c96a00 x1876699512700928/t4294973869(4294973869) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760782 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760766.611559:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760766.611564:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760766.611570:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760766.611589:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760766.611605:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512700928:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760766.611728:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6550, 6550] 00000080:00200000:1.0:1789760766.611739:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760766.611743:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760766.611752:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26832896 i_blocks 53255 00000020:00200000:1.0:1789760766.611754:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760766.611761:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760766.611784:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26832896, count: 4096 00000080:00200000:1.0:1789760766.611789:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26832896, bytes: 4096 00000080:00200000:1.0:1789760766.611797:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760766.611804:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760766.611806:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26832896, 26836992) 0 0 00000080:00200000:1.0:1789760766.611810:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760766.611823:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6551, 6551] 00020000:00200000:1.0:1789760766.611830:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26832896 chunk: [26832896, 26836992) 26836992, 4096 00020000:00200000:1.0:1789760766.611833:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760766.611848:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6551, 6551] 00000008:00010000:1.0:1789760766.612036:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760766.612042:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760766.612052:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26832896, 26836992) append 0 append lockpos 0 00000080:00200000:1.0:1789760766.612064:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26832896=0x1997000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760766.612113:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760766.612116:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26836992 00000020:00010000:1.0:1789760766.612124:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760766.612127:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.612130:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760766.612133:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760766.612135:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760766.612142:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760766.612149:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760766.612165:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760766.614221:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94380 - object 0x240000400:5 offset 26832896<>26836992 00000100:00100000:0.0:1789760766.614306:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512701056:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760766.668115:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973870 gen 3 00000008:00010000:0.0:1789760766.668164:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760766.668172:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760766.668179:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760766.668205:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760766.668230:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512701056:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760766.668256:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6551, 6551] 00000080:00200000:2.0:1789760766.668269:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.668274:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.668287:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26836992 i_blocks 53255 00000020:00200000:2.0:1789760766.668291:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.668299:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.668330:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26836992, count: 4096 00000080:00200000:2.0:1789760766.668336:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26836992, bytes: 4096 00000080:00200000:2.0:1789760766.668344:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.668351:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.668353:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26836992, 26841088) 0 0 00000080:00200000:2.0:1789760766.668356:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.668368:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6552, 6552] 00020000:00200000:2.0:1789760766.668375:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26836992 chunk: [26836992, 26841088) 26841088, 4096 00020000:00200000:2.0:1789760766.668378:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.668394:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6552, 6552] 00000008:00010000:2.0:1789760766.668544:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.668550:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760766.668560:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26836992, 26841088) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.668570:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26836992=0x1998000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.668610:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.668612:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26841088 00000020:00010000:2.0:1789760766.668617:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.668620:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.668622:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.668624:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.668626:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.668632:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.668637:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760766.668652:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760766.668738:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 26836992<>26841088 00000100:00100000:3.0:1789760766.668812:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512701184:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760766.713567:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973871 gen 3 00000008:00010000:3.0:1789760766.713608:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760766.713615:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760766.713624:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760766.713650:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760766.713678:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512701184:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760766.713928:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6552, 6552] 00000080:00200000:2.0:1789760766.713944:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.713949:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.713966:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26841088 i_blocks 53255 00000020:00200000:2.0:1789760766.713969:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.713977:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.714011:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26841088, count: 4096 00000080:00200000:2.0:1789760766.714030:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26841088, bytes: 4096 00000080:00200000:2.0:1789760766.714041:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.714049:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.714051:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26841088, 26845184) 0 0 00000080:00200000:2.0:1789760766.714055:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.714073:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6553, 6553] 00020000:00200000:2.0:1789760766.714081:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26841088 chunk: [26841088, 26845184) 26845184, 4096 00020000:00200000:2.0:1789760766.714085:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.714106:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6553, 6553] 00000008:00010000:2.0:1789760766.714128:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.714134:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760766.714147:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26841088, 26845184) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.714161:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26841088=0x1999000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.714217:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.714220:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26845184 00000020:00010000:2.0:1789760766.714227:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.714231:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.714243:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.714247:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.714250:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.714257:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.714263:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760766.714276:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760766.714358:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 26841088<>26845184 00000100:00100000:3.0:1789760766.714500:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512701312:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760766.749221:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760766.760176:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973872 gen 3 00000008:00010000:3.0:1789760766.760214:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760766.760220:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760766.760246:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760766.760263:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760766.760287:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512701312:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760766.762154:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6553, 6553] 00000080:00200000:2.0:1789760766.762171:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.762177:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.762191:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26845184 i_blocks 53255 00000020:00200000:2.0:1789760766.762196:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.762208:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.762248:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26845184, count: 4096 00000080:00200000:2.0:1789760766.762255:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26845184, bytes: 4096 00000080:00200000:2.0:1789760766.762264:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.762272:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.762274:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26845184, 26849280) 0 0 00000080:00200000:2.0:1789760766.762278:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.762293:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6554, 6554] 00020000:00200000:2.0:1789760766.762300:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26845184 chunk: [26845184, 26849280) 26849280, 4096 00020000:00200000:2.0:1789760766.762304:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.762319:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6554, 6554] 00000008:00010000:2.0:1789760766.762333:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.762337:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760766.762346:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26845184, 26849280) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.762356:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26845184=0x199a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.762406:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.762409:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26849280 00000020:00010000:2.0:1789760766.762416:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.762420:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.762422:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.762798:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.762801:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.762810:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.762817:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760766.762835:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760766.763095:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 26845184<>26849280 00000100:00100000:3.0:1789760766.763264:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760766.763285:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512701440:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760766.804704:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973873 gen 3 00000008:00010000:3.0:1789760766.804741:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760766.804746:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760766.804754:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760766.804774:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760766.804797:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512701440:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760766.804827:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6554, 6554] 00000080:00200000:2.0:1789760766.804841:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.804847:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.804869:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26849280 i_blocks 53255 00000020:00200000:2.0:1789760766.804873:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.804881:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.804933:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26849280, count: 4096 00000080:00200000:2.0:1789760766.804940:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26849280, bytes: 4096 00000080:00200000:2.0:1789760766.804948:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.804955:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.804957:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26849280, 26853376) 0 0 00000080:00200000:2.0:1789760766.804961:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.804973:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6555, 6555] 00020000:00200000:2.0:1789760766.804980:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26849280 chunk: [26849280, 26853376) 26853376, 4096 00020000:00200000:2.0:1789760766.804986:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.805001:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6555, 6555] 00000008:00010000:2.0:1789760766.805016:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.805021:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760766.805031:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26849280, 26853376) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.805041:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26849280=0x199b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.805086:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.805088:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26853376 00000020:00010000:2.0:1789760766.805096:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.805099:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.805102:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.805105:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.805108:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.805114:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.805120:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760766.805133:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760766.805244:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 26849280<>26853376 00000100:00100000:3.0:1789760766.805333:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512701568:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760766.840165:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760766.840178:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760766.840188:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760766.840209:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760766.840244:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512701568:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760766.840359:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6555, 6555] 00000080:00200000:2.0:1789760766.840372:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.840377:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.840389:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26853376 i_blocks 53255 00000020:00200000:2.0:1789760766.840392:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.840401:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.840432:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26853376, count: 4096 00000080:00200000:2.0:1789760766.840439:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26853376, bytes: 4096 00000080:00200000:2.0:1789760766.840447:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.840454:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.840456:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26853376, 26857472) 0 0 00000080:00200000:2.0:1789760766.840459:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.840474:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6556, 6556] 00020000:00200000:2.0:1789760766.840482:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26853376 chunk: [26853376, 26857472) 26857472, 4096 00020000:00200000:2.0:1789760766.840486:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.840502:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6556, 6556] 00000008:00010000:2.0:1789760766.840516:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.840521:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760766.840531:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26853376, 26857472) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.840543:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26853376=0x199c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.840588:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.840591:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26857472 00000020:00010000:2.0:1789760766.840597:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.840599:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.840602:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.840605:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.840608:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.840614:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.840619:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760766.840646:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760766.840725:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 26853376<>26857472 00000100:00100000:3.0:1789760766.840813:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512701696:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760766.880204:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973874 gen 3 00000100:00100000:3.0:1789760766.880244:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368d880 x1876699512701696/t4294973875(4294973875) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760782 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760766.880285:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760766.880291:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760766.880299:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760766.880318:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760766.880339:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512701696:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760766.880418:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6556, 6556] 00000080:00200000:2.0:1789760766.880430:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.880435:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.880447:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26857472 i_blocks 53255 00000020:00200000:2.0:1789760766.880450:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.880460:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.880487:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26857472, count: 4096 00000080:00200000:2.0:1789760766.880494:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26857472, bytes: 4096 00000080:00200000:2.0:1789760766.880503:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.880511:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.880514:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26857472, 26861568) 0 0 00000080:00200000:2.0:1789760766.880518:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.880533:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6557, 6557] 00020000:00200000:2.0:1789760766.880540:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26857472 chunk: [26857472, 26861568) 26861568, 4096 00020000:00200000:2.0:1789760766.880545:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.880561:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6557, 6557] 00000008:00010000:2.0:1789760766.880575:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.880580:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760766.880589:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26857472, 26861568) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.880600:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26857472=0x199d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.880657:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.880660:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26861568 00000020:00010000:2.0:1789760766.880667:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.880670:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.880673:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.880676:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.880679:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.880685:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.880690:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760766.880713:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760766.880790:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 26857472<>26861568 00000100:00100000:3.0:1789760766.881027:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512701824:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760766.917293:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973876 gen 3 00000008:00010000:3.0:1789760766.917340:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760766.917346:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760766.917354:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760766.917373:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760766.917395:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512701824:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760766.917481:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6557, 6557] 00000080:00200000:2.0:1789760766.917494:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.917499:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.917512:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26861568 i_blocks 53255 00000020:00200000:2.0:1789760766.917516:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.917524:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.917551:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26861568, count: 4096 00000080:00200000:2.0:1789760766.917557:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26861568, bytes: 4096 00000080:00200000:2.0:1789760766.917564:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.917572:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.917574:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26861568, 26865664) 0 0 00000080:00200000:2.0:1789760766.917578:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.917592:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6558, 6558] 00020000:00200000:2.0:1789760766.917599:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26861568 chunk: [26861568, 26865664) 26865664, 4096 00020000:00200000:2.0:1789760766.917602:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.917615:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6558, 6558] 00000008:00010000:2.0:1789760766.917627:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.917631:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760766.917640:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26861568, 26865664) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.917649:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26861568=0x199e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.917690:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.917692:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26865664 00000020:00010000:2.0:1789760766.917697:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.917700:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.917702:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.917705:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.917707:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.917712:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.917716:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760766.917728:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760766.917802:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 26861568<>26865664 00000100:00100000:3.0:1789760766.917896:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512701952:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760766.954741:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973877 gen 3 00000008:00010000:3.0:1789760766.954776:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760766.954783:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760766.954809:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760766.954825:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760766.954848:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512701952:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760766.954959:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6558, 6558] 00000080:00200000:2.0:1789760766.954973:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760766.954978:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760766.954988:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26865664 i_blocks 53255 00000020:00200000:2.0:1789760766.954992:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760766.954999:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760766.955024:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26865664, count: 4096 00000080:00200000:2.0:1789760766.955029:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26865664, bytes: 4096 00000080:00200000:2.0:1789760766.955038:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760766.955044:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760766.955046:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26865664, 26869760) 0 0 00000080:00200000:2.0:1789760766.955050:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760766.955063:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6559, 6559] 00020000:00200000:2.0:1789760766.955069:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26865664 chunk: [26865664, 26869760) 26869760, 4096 00020000:00200000:2.0:1789760766.955073:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760766.955089:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6559, 6559] 00000008:00010000:2.0:1789760766.955102:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760766.955106:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760766.955114:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26865664, 26869760) append 0 append lockpos 0 00000080:00200000:2.0:1789760766.955124:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26865664=0x199f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760766.955164:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760766.955166:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26869760 00000020:00010000:2.0:1789760766.955173:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760766.955176:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.955179:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760766.955183:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760766.955186:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760766.955192:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760766.955196:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760766.955208:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760766.955301:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 26865664<>26869760 00000100:00100000:3.0:1789760766.955491:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512702080:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760767.033777:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973878 gen 3 00000008:00010000:3.0:1789760767.033825:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760767.033832:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.033841:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.033861:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760767.033888:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512702080:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760767.034062:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6559, 6559] 00000080:00200000:2.0:1789760767.034074:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.034079:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.034090:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26869760 i_blocks 53255 00000020:00200000:2.0:1789760767.034093:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.034101:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.034124:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26869760, count: 4096 00000080:00200000:2.0:1789760767.034129:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26869760, bytes: 4096 00000080:00200000:2.0:1789760767.034137:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.034144:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.034146:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26869760, 26873856) 0 0 00000080:00200000:2.0:1789760767.034149:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.034170:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6560, 6560] 00020000:00200000:2.0:1789760767.034179:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26869760 chunk: [26869760, 26873856) 26873856, 4096 00020000:00200000:2.0:1789760767.034183:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.034200:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6560, 6560] 00000008:00010000:2.0:1789760767.034213:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.034218:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760767.034227:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26869760, 26873856) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.034251:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26869760=0x19a0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.034294:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.034296:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26873856 00000020:00010000:2.0:1789760767.034303:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.034306:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.034309:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.034312:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.034315:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.034321:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.034327:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.034340:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760767.034415:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 26869760<>26873856 00000100:00100000:3.0:1789760767.034517:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512702208:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760767.072616:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973879 gen 3 00000008:00010000:3.0:1789760767.072669:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760767.072676:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.072685:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.072705:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760767.072729:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512702208:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760767.072881:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6560, 6560] 00000080:00200000:2.0:1789760767.072894:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.072932:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.072943:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26873856 i_blocks 53255 00000020:00200000:2.0:1789760767.072946:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.072954:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.072980:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26873856, count: 4096 00000080:00200000:2.0:1789760767.072987:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26873856, bytes: 4096 00000080:00200000:2.0:1789760767.073180:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.073188:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.073190:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26873856, 26877952) 0 0 00000080:00200000:2.0:1789760767.073194:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.073207:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6561, 6561] 00020000:00200000:2.0:1789760767.073213:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26873856 chunk: [26873856, 26877952) 26877952, 4096 00020000:00200000:2.0:1789760767.073217:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.073242:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6561, 6561] 00000008:00010000:2.0:1789760767.073257:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.073262:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760767.073272:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26873856, 26877952) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.073283:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26873856=0x19a1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.073330:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.073333:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26877952 00000020:00010000:2.0:1789760767.073339:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.073343:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.073345:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.073349:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.073351:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.073358:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.073364:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.073380:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760767.073461:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 26873856<>26877952 00000100:00100000:3.0:1789760767.073559:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512702336:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760767.112880:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760767.112892:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.112929:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.112949:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760767.112973:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512702336:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760767.113018:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6561, 6561] 00000080:00200000:2.0:1789760767.113029:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.113035:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.113050:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26877952 i_blocks 53255 00000020:00200000:2.0:1789760767.113053:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.113061:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.113086:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26877952, count: 4096 00000080:00200000:2.0:1789760767.113092:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26877952, bytes: 4096 00000080:00200000:2.0:1789760767.113100:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.113107:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.113109:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26877952, 26882048) 0 0 00000080:00200000:2.0:1789760767.113113:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.113125:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6562, 6562] 00020000:00200000:2.0:1789760767.113131:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26877952 chunk: [26877952, 26882048) 26882048, 4096 00020000:00200000:2.0:1789760767.113135:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.113149:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6562, 6562] 00000008:00010000:2.0:1789760767.113172:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.113177:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760767.113186:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26877952, 26882048) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.113197:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26877952=0x19a2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.113321:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.113324:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26882048 00000020:00010000:2.0:1789760767.113332:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.113407:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.113411:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.113414:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.113416:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.113425:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.113431:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.113446:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760767.113526:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 26877952<>26882048 00000100:00100000:3.0:1789760767.113625:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512702464:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760767.142822:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973881 gen 3 00000008:00010000:3.0:1789760767.142873:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760767.142879:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.142887:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.142936:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760767.142960:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512702464:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760767.143042:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6562, 6562] 00000080:00200000:2.0:1789760767.143055:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.143061:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.143073:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26882048 i_blocks 53255 00000020:00200000:2.0:1789760767.143077:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.143085:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.143115:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26882048, count: 4096 00000080:00200000:2.0:1789760767.143121:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26882048, bytes: 4096 00000080:00200000:2.0:1789760767.143129:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.143137:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.143139:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26882048, 26886144) 0 0 00000080:00200000:2.0:1789760767.143143:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.143156:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6563, 6563] 00020000:00200000:2.0:1789760767.143163:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26882048 chunk: [26882048, 26886144) 26886144, 4096 00020000:00200000:2.0:1789760767.143167:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.143182:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6563, 6563] 00000008:00010000:2.0:1789760767.143195:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.143200:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760767.143209:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26882048, 26886144) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.143219:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26882048=0x19a3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.143268:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.143271:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26886144 00000020:00010000:2.0:1789760767.143277:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.143280:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.143283:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.143286:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.143289:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.143294:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.143300:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.143313:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760767.143407:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 26882048<>26886144 00000100:00100000:3.0:1789760767.143554:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512702592:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760767.180791:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973882 gen 3 00000008:00010000:3.0:1789760767.180826:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760767.180832:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.180841:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.180857:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760767.180888:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512702592:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760767.181016:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6563, 6563] 00000080:00200000:2.0:1789760767.181028:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.181033:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.181044:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26886144 i_blocks 53255 00000020:00200000:2.0:1789760767.181047:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.181054:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.181078:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26886144, count: 4096 00000080:00200000:2.0:1789760767.181084:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26886144, bytes: 4096 00000080:00200000:2.0:1789760767.181092:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.181099:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.181101:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26886144, 26890240) 0 0 00000080:00200000:2.0:1789760767.181104:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.181116:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6564, 6564] 00020000:00200000:2.0:1789760767.181122:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26886144 chunk: [26886144, 26890240) 26890240, 4096 00020000:00200000:2.0:1789760767.181126:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.181141:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6564, 6564] 00000008:00010000:2.0:1789760767.181154:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.181158:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760767.181167:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26886144, 26890240) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.181177:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26886144=0x19a4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.181219:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.181222:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26890240 00000020:00010000:2.0:1789760767.181229:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.181244:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.181247:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.181250:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.181253:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.181259:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.181264:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.181278:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760767.181364:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 26886144<>26890240 00000100:00100000:3.0:1789760767.181586:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512702720:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760767.210271:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973883 gen 3 00000008:00010000:3.0:1789760767.210306:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760767.210313:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.210321:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.210382:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760767.210405:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512702720:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760767.210572:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6564, 6564] 00000080:00200000:2.0:1789760767.210584:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.210589:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.210600:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26890240 i_blocks 53255 00000020:00200000:2.0:1789760767.210603:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.210610:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.210635:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26890240, count: 4096 00000080:00200000:2.0:1789760767.210641:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26890240, bytes: 4096 00000080:00200000:2.0:1789760767.210722:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.210729:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.210733:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26890240, 26894336) 0 0 00000080:00200000:2.0:1789760767.210737:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.210749:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6565, 6565] 00020000:00200000:2.0:1789760767.210756:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26890240 chunk: [26890240, 26894336) 26894336, 4096 00020000:00200000:2.0:1789760767.210760:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.210775:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6565, 6565] 00000008:00010000:2.0:1789760767.210789:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.210794:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760767.210803:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26890240, 26894336) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.210813:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26890240=0x19a5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.211175:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.211177:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26894336 00000020:00010000:2.0:1789760767.211185:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.211188:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.211191:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.211194:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.211197:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.211203:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.211209:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.211224:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760767.211306:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 26890240<>26894336 00000100:00100000:3.0:1789760767.211419:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512702848:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760767.254168:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973884 gen 3 00000008:00010000:3.0:1789760767.254207:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760767.254214:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.254223:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.254247:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760767.254270:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512702848:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760767.254504:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6565, 6565] 00000080:00200000:2.0:1789760767.254517:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.254522:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.254536:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26894336 i_blocks 53255 00000020:00200000:2.0:1789760767.254540:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.254548:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.254573:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26894336, count: 4096 00000080:00200000:2.0:1789760767.254579:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26894336, bytes: 4096 00000080:00200000:2.0:1789760767.254587:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.254594:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.254596:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26894336, 26898432) 0 0 00000080:00200000:2.0:1789760767.254599:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.254611:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6566, 6566] 00020000:00200000:2.0:1789760767.254617:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26894336 chunk: [26894336, 26898432) 26898432, 4096 00020000:00200000:2.0:1789760767.254630:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.254649:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6566, 6566] 00000008:00010000:2.0:1789760767.254663:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.254668:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760767.254678:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26894336, 26898432) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.254690:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26894336=0x19a6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.254736:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.254738:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26898432 00000020:00010000:2.0:1789760767.254745:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.254748:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.254750:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.254752:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.254755:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.254760:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.254765:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.254778:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000008:00100000:2.0:1789760767.254853:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 26894336<>26898432 00000100:00100000:3.0:1789760767.255059:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512702976:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760767.306314:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973885 gen 3 00000008:00010000:3.0:1789760767.306351:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760767.306358:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.306366:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.306384:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760767.306408:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512702976:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760767.306495:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6566, 6566] 00000080:00200000:2.0:1789760767.306507:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.306513:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.306525:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26898432 i_blocks 53255 00000020:00200000:2.0:1789760767.306528:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.306536:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.306561:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26898432, count: 4096 00000080:00200000:2.0:1789760767.306567:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26898432, bytes: 4096 00000080:00200000:2.0:1789760767.306575:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.306582:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.306584:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26898432, 26902528) 0 0 00000080:00200000:2.0:1789760767.306588:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.306601:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6567, 6567] 00020000:00200000:2.0:1789760767.306608:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26898432 chunk: [26898432, 26902528) 26902528, 4096 00020000:00200000:2.0:1789760767.306612:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.306636:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6567, 6567] 00000008:00010000:2.0:1789760767.306650:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.306655:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760767.306664:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26898432, 26902528) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.306674:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26898432=0x19a7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.306716:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.306718:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26902528 00000020:00010000:2.0:1789760767.306725:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.306729:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.306732:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.306735:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.306738:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.306744:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.306749:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.306762:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760767.306835:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 26898432<>26902528 00000100:00100000:3.0:1789760767.306922:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760767.306936:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512703104:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760767.345778:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973886 gen 3 00000008:00010000:3.0:1789760767.345812:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760767.345818:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.345827:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.345931:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760767.345954:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512703104:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760767.349197:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6567, 6567] 00000080:00200000:2.0:1789760767.349216:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.349222:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.349246:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26902528 i_blocks 53255 00000020:00200000:2.0:1789760767.349252:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.349264:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.349289:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26902528, count: 4096 00000080:00200000:2.0:1789760767.349296:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26902528, bytes: 4096 00000080:00200000:2.0:1789760767.349308:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.349316:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.349318:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26902528, 26906624) 0 0 00000080:00200000:2.0:1789760767.349322:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.349335:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6568, 6568] 00020000:00200000:2.0:1789760767.349341:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26902528 chunk: [26902528, 26906624) 26906624, 4096 00020000:00200000:2.0:1789760767.349345:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.349361:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6568, 6568] 00000008:00010000:2.0:1789760767.349375:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.349380:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760767.349389:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26902528, 26906624) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.349401:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26902528=0x19a8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.349453:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.349456:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26906624 00000020:00010000:2.0:1789760767.349463:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.349466:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.349468:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.349471:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.349473:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.349481:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.349486:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.349509:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760767.349587:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 26902528<>26906624 00000100:00100000:3.0:1789760767.349670:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512703232:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760767.396375:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760767.396386:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.396395:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.396482:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760767.396506:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512703232:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760767.399127:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6568, 6568] 00000080:00200000:2.0:1789760767.399146:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.399151:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.399164:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26906624 i_blocks 53255 00000020:00200000:2.0:1789760767.399168:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.399180:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.399205:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26906624, count: 4096 00000080:00200000:2.0:1789760767.399212:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26906624, bytes: 4096 00000080:00200000:2.0:1789760767.399224:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.399242:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.399244:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26906624, 26910720) 0 0 00000080:00200000:2.0:1789760767.399247:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.399261:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6569, 6569] 00020000:00200000:2.0:1789760767.399268:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26906624 chunk: [26906624, 26910720) 26910720, 4096 00020000:00200000:2.0:1789760767.399272:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.399289:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6569, 6569] 00000008:00010000:2.0:1789760767.399304:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.399309:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760767.399318:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26906624, 26910720) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.399329:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26906624=0x19a9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.399378:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.399381:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26910720 00000020:00010000:2.0:1789760767.399387:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.399390:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.399393:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.399396:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.399399:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.399406:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.399412:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.399434:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000008:00100000:2.0:1789760767.399520:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 26906624<>26910720 00000100:00100000:3.0:1789760767.399603:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512703360:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760767.429278:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973888 gen 3 00000008:00010000:3.0:1789760767.429387:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760767.429394:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.429401:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.429420:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760767.429443:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512703360:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760767.429528:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6569, 6569] 00000080:00200000:2.0:1789760767.429540:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.429545:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.429557:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26910720 i_blocks 53255 00000020:00200000:2.0:1789760767.429560:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.429568:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.429592:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26910720, count: 4096 00000080:00200000:2.0:1789760767.429598:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26910720, bytes: 4096 00000080:00200000:2.0:1789760767.429605:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.429613:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.429615:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26910720, 26914816) 0 0 00000080:00200000:2.0:1789760767.429618:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.429631:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6570, 6570] 00020000:00200000:2.0:1789760767.429637:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26910720 chunk: [26910720, 26914816) 26914816, 4096 00020000:00200000:2.0:1789760767.429641:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.429656:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6570, 6570] 00000008:00010000:2.0:1789760767.429669:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.429674:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760767.429683:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26910720, 26914816) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.429694:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26910720=0x19aa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.429738:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.429741:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26914816 00000020:00010000:2.0:1789760767.429747:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.429750:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.429753:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.429756:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.429760:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.429766:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.429771:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.429783:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000008:00100000:2.0:1789760767.429857:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 26910720<>26914816 00000100:00100000:3.0:1789760767.430067:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760767.430081:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512703488:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760767.477176:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973889 gen 3 00000008:00010000:3.0:1789760767.477209:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760767.477215:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.477222:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.477245:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760767.477268:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512703488:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760767.477414:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6570, 6570] 00000080:00200000:2.0:1789760767.477427:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.477432:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.477443:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26914816 i_blocks 53255 00000020:00200000:2.0:1789760767.477447:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.477455:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.477482:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26914816, count: 4096 00000080:00200000:2.0:1789760767.477489:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26914816, bytes: 4096 00000080:00200000:2.0:1789760767.477497:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.477504:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.477506:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26914816, 26918912) 0 0 00000080:00200000:2.0:1789760767.477510:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.477523:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6571, 6571] 00020000:00200000:2.0:1789760767.477530:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26914816 chunk: [26914816, 26918912) 26918912, 4096 00020000:00200000:2.0:1789760767.477534:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.477549:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6571, 6571] 00000008:00010000:2.0:1789760767.477564:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.477569:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760767.477577:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26914816, 26918912) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.477587:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26914816=0x19ab000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.477631:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.477634:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26918912 00000020:00010000:2.0:1789760767.477641:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.477644:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.477647:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.477650:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.477652:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.477659:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.477664:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.477677:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760767.477750:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 26914816<>26918912 00000100:00100000:3.0:1789760767.478128:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512703616:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760767.513760:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973890 gen 3 00000008:00010000:3.0:1789760767.513796:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760767.513802:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.513810:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.513869:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760767.513936:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512703616:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760767.514217:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6571, 6571] 00000080:00200000:2.0:1789760767.514239:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.514245:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.514256:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26918912 i_blocks 53255 00000020:00200000:2.0:1789760767.514260:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.514268:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.514421:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26918912, count: 4096 00000080:00200000:2.0:1789760767.514428:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26918912, bytes: 4096 00000080:00200000:2.0:1789760767.514436:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.514447:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.514450:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26918912, 26923008) 0 0 00000080:00200000:2.0:1789760767.514454:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.514468:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6572, 6572] 00020000:00200000:2.0:1789760767.514475:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26918912 chunk: [26918912, 26923008) 26923008, 4096 00020000:00200000:2.0:1789760767.514479:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.514492:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6572, 6572] 00000008:00010000:2.0:1789760767.514507:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.514513:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760767.514522:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26918912, 26923008) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.514533:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26918912=0x19ac000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.514575:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.514578:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26923008 00000020:00010000:2.0:1789760767.514584:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.514588:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.514591:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.514594:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.514597:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.514603:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.514608:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.514693:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760767.514771:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 26918912<>26923008 00000100:00100000:3.0:1789760767.514833:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512703744:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760767.582942:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973891 gen 3 00000008:00010000:3.0:1789760767.582980:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760767.582996:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.583004:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.583020:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760767.583042:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512703744:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760767.583182:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6572, 6572] 00000080:00200000:2.0:1789760767.583194:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.583199:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.583211:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26923008 i_blocks 53255 00000020:00200000:2.0:1789760767.583214:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.583223:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.583257:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26923008, count: 4096 00000080:00200000:2.0:1789760767.583263:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26923008, bytes: 4096 00000080:00200000:2.0:1789760767.583271:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.583279:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.583281:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26923008, 26927104) 0 0 00000080:00200000:2.0:1789760767.583284:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.583297:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6573, 6573] 00020000:00200000:2.0:1789760767.583303:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26923008 chunk: [26923008, 26927104) 26927104, 4096 00020000:00200000:2.0:1789760767.583308:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.583341:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6573, 6573] 00000008:00010000:2.0:1789760767.583356:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.583361:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760767.583371:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26923008, 26927104) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.583382:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26923008=0x19ad000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.583427:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.583429:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26927104 00000020:00010000:2.0:1789760767.583436:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.583440:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.583443:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.583447:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.583449:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.583457:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.583462:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.583476:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000008:00100000:2.0:1789760767.583556:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 26923008<>26927104 00000100:00100000:3.0:1789760767.583819:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760767.583835:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512703872:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760767.631811:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760767.631823:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.631833:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.631851:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760767.631875:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512703872:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760767.632169:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6573, 6573] 00000080:00200000:2.0:1789760767.632182:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.632186:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.632197:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26927104 i_blocks 53255 00000020:00200000:2.0:1789760767.632201:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.632208:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.632244:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26927104, count: 4096 00000080:00200000:2.0:1789760767.632250:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26927104, bytes: 4096 00000080:00200000:2.0:1789760767.632257:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.632264:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.632266:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26927104, 26931200) 0 0 00000080:00200000:2.0:1789760767.632270:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.632283:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6574, 6574] 00020000:00200000:2.0:1789760767.632289:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26927104 chunk: [26927104, 26931200) 26931200, 4096 00020000:00200000:2.0:1789760767.632293:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.632309:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6574, 6574] 00000008:00010000:2.0:1789760767.632323:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.632327:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760767.632335:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26927104, 26931200) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.632346:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26927104=0x19ae000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.632389:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.632392:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26931200 00000020:00010000:2.0:1789760767.632398:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.632402:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.632404:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.632408:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.632410:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.632416:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.632421:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.632434:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760767.632515:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 26927104<>26931200 00000100:00100000:3.0:1789760767.632615:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512704000:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760767.668795:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973893 gen 3 00000008:00010000:3.0:1789760767.668835:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760767.668841:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.668847:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.668925:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760767.668957:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512704000:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760767.669312:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6574, 6574] 00000080:00200000:2.0:1789760767.669326:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.669331:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.669343:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26931200 i_blocks 53255 00000020:00200000:2.0:1789760767.669346:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.669354:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.669379:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26931200, count: 4096 00000080:00200000:2.0:1789760767.669385:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26931200, bytes: 4096 00000080:00200000:2.0:1789760767.669393:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.669400:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.669402:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26931200, 26935296) 0 0 00000080:00200000:2.0:1789760767.669406:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.669421:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6575, 6575] 00020000:00200000:2.0:1789760767.669427:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26931200 chunk: [26931200, 26935296) 26935296, 4096 00020000:00200000:2.0:1789760767.669432:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.669449:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6575, 6575] 00000008:00010000:2.0:1789760767.669462:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.669467:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760767.669475:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26931200, 26935296) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.669486:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26931200=0x19af000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.669532:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.669535:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26935296 00000020:00010000:2.0:1789760767.669541:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.669544:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.669547:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.669550:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.669552:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.669559:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.669565:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.669579:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760767.669654:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 26931200<>26935296 00000100:00100000:3.0:1789760767.669729:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512704128:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760767.720470:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973894 gen 3 00000008:00010000:3.0:1789760767.720507:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760767.720514:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.720522:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.720594:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760767.720632:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512704128:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760767.724081:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6575, 6575] 00000080:00200000:2.0:1789760767.724101:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760767.724107:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760767.724120:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26935296 i_blocks 53255 00000020:00200000:2.0:1789760767.724126:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760767.724139:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760767.724169:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26935296, count: 4096 00000080:00200000:2.0:1789760767.724176:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26935296, bytes: 4096 00000080:00200000:2.0:1789760767.724188:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760767.724196:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760767.724198:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26935296, 26939392) 0 0 00000080:00200000:2.0:1789760767.724202:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760767.724683:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6576, 6576] 00020000:00200000:2.0:1789760767.724693:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26935296 chunk: [26935296, 26939392) 26939392, 4096 00020000:00200000:2.0:1789760767.724698:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760767.724716:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6576, 6576] 00000008:00010000:2.0:1789760767.724733:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760767.724738:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760767.724748:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26935296, 26939392) append 0 append lockpos 0 00000080:00200000:2.0:1789760767.724759:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26935296=0x19b0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760767.724815:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760767.724818:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26939392 00000020:00010000:2.0:1789760767.724826:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760767.724829:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.724832:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760767.724835:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760767.724838:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760767.724845:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760767.724850:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760767.724872:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760767.726186:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 26935296<>26939392 00000100:00100000:3.0:1789760767.726251:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760767.728559:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512704256:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760767.774318:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760767.784039:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973895 gen 3 00000008:00010000:3.0:1789760767.784073:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760767.784079:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.784087:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.784098:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760767.784119:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512704256:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760767.784276:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6576, 6576] 00000080:00200000:0.0:1789760767.784289:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760767.784294:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760767.784304:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26939392 i_blocks 53255 00000020:00200000:0.0:1789760767.784307:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760767.784315:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760767.784341:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26939392, count: 4096 00000080:00200000:0.0:1789760767.784348:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26939392, bytes: 4096 00000080:00200000:0.0:1789760767.784355:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760767.784363:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760767.784364:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26939392, 26943488) 0 0 00000080:00200000:0.0:1789760767.784369:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760767.784380:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6577, 6577] 00020000:00200000:0.0:1789760767.784387:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26939392 chunk: [26939392, 26943488) 26943488, 4096 00020000:00200000:0.0:1789760767.784391:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760767.784406:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6577, 6577] 00000008:00010000:0.0:1789760767.784431:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760767.784436:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760767.784447:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26939392, 26943488) append 0 append lockpos 0 00000080:00200000:0.0:1789760767.784457:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26939392=0x19b1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760767.784500:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760767.784503:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26943488 00000020:00010000:0.0:1789760767.784508:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760767.784511:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.784514:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.784517:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760767.784519:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760767.784525:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760767.784530:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760767.784543:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc240 00000008:00100000:0.0:1789760767.784629:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149180 - object 0x240000400:5 offset 26939392<>26943488 00000100:00100000:1.0:1789760767.784852:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512704384:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760767.828499:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973896 gen 3 00000008:00010000:1.0:1789760767.828540:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760767.828546:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760767.828555:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760767.828574:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760767.828598:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512704384:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760767.828611:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6577, 6577] 00000080:00200000:3.0:1789760767.828747:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760767.828753:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760767.828765:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26943488 i_blocks 53255 00000020:00200000:3.0:1789760767.828768:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760767.828777:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760767.828803:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26943488, count: 4096 00000080:00200000:3.0:1789760767.828809:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26943488, bytes: 4096 00000080:00200000:3.0:1789760767.828816:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760767.828823:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760767.828825:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26943488, 26947584) 0 0 00000080:00200000:3.0:1789760767.828829:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760767.828840:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6578, 6578] 00020000:00200000:3.0:1789760767.828846:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26943488 chunk: [26943488, 26947584) 26947584, 4096 00020000:00200000:3.0:1789760767.828850:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760767.828865:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6578, 6578] 00000008:00010000:3.0:1789760767.828879:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760767.828884:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760767.828892:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26943488, 26947584) append 0 append lockpos 0 00000080:00200000:3.0:1789760767.828987:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26943488=0x19b2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760767.829033:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760767.829035:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26947584 00000020:00010000:3.0:1789760767.829041:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760767.829044:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.829046:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760767.829050:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760767.829052:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760767.829057:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760767.829062:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760767.829075:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760767.829146:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad500 - object 0x240000400:5 offset 26943488<>26947584 00000100:00100000:2.0:1789760767.829221:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512704512:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760767.883891:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:3.0:1789760767.884937:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760767.884949:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760767.885320:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:3.0:1789760767.885349:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512704512:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760767.885952:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6578, 6578] 00000080:00200000:1.0:1789760767.885966:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760767.885971:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760767.885984:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26947584 i_blocks 53255 00000020:00200000:1.0:1789760767.885988:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760767.885997:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760767.886023:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26947584, count: 4096 00000080:00200000:1.0:1789760767.886029:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26947584, bytes: 4096 00000080:00200000:1.0:1789760767.886041:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760767.886047:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760767.886049:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26947584, 26951680) 0 0 00000080:00200000:1.0:1789760767.886053:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760767.886066:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6579, 6579] 00020000:00200000:1.0:1789760767.886073:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26947584 chunk: [26947584, 26951680) 26951680, 4096 00020000:00200000:1.0:1789760767.886077:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760767.886093:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6579, 6579] 00000008:00010000:1.0:1789760767.886107:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760767.886111:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760767.886120:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26947584, 26951680) append 0 append lockpos 0 00000080:00200000:1.0:1789760767.886130:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26947584=0x19b3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760767.888076:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760767.888086:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26951680 00000020:00010000:1.0:1789760767.888098:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760767.888103:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760767.888106:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760767.888110:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760767.888113:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760767.888124:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760767.888132:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760767.888161:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760767.888274:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 26947584<>26951680 00000100:00100000:1.0:1789760767.888326:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760767.888346:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512704640:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760767.959451:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973898 gen 3 00000008:00010000:1.0:1789760767.959495:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760767.959501:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760767.959509:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760767.959527:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760767.959550:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512704640:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760767.959636:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6579, 6579] 00000080:00200000:0.0:1789760767.959648:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760767.959653:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760767.959664:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26951680 i_blocks 53255 00000020:00200000:0.0:1789760767.959667:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760767.959674:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760767.959699:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26951680, count: 4096 00000080:00200000:0.0:1789760767.959705:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26951680, bytes: 4096 00000080:00200000:0.0:1789760767.959712:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760767.959718:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760767.959719:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26951680, 26955776) 0 0 00000080:00200000:0.0:1789760767.959722:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760767.959794:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6580, 6580] 00020000:00200000:0.0:1789760767.959802:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26951680 chunk: [26951680, 26955776) 26955776, 4096 00020000:00200000:0.0:1789760767.959806:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760767.959958:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6580, 6580] 00000008:00010000:0.0:1789760767.959975:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760767.959979:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760767.959989:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26951680, 26955776) append 0 append lockpos 0 00000080:00200000:0.0:1789760767.960000:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26951680=0x19b4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760767.960049:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760767.960133:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26955776 00000020:00010000:0.0:1789760767.960141:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760767.960145:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.960148:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760767.960151:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760767.960153:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760767.960160:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760767.960165:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760767.960180:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc480 00000008:00100000:0.0:1789760767.960279:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149c00 - object 0x240000400:5 offset 26951680<>26955776 00000100:00100000:0.0:1789760767.960319:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760767.960333:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512704768:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760768.008954:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973899 gen 3 00000008:00010000:0.0:1789760768.008988:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:0.0:1789760768.008994:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760768.009003:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760768.009074:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:0.0:1789760768.009098:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512704768:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760768.010003:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6580, 6580] 00000080:00200000:3.0:1789760768.010071:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760768.010077:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760768.010089:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26955776 i_blocks 53255 00000020:00200000:3.0:1789760768.010093:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760768.010103:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760768.010219:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26955776, count: 4096 00000080:00200000:3.0:1789760768.010236:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26955776, bytes: 4096 00000080:00200000:3.0:1789760768.010246:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760768.010253:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760768.010255:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26955776, 26959872) 0 0 00000080:00200000:3.0:1789760768.010258:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760768.010272:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6581, 6581] 00020000:00200000:3.0:1789760768.010279:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26955776 chunk: [26955776, 26959872) 26959872, 4096 00020000:00200000:3.0:1789760768.010282:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760768.010298:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6581, 6581] 00000008:00010000:3.0:1789760768.012768:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760768.012783:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760768.012798:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26955776, 26959872) append 0 append lockpos 0 00000080:00200000:3.0:1789760768.012815:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26955776=0x19b5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760768.012873:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760768.012876:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26959872 00000020:00010000:3.0:1789760768.012884:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760768.012887:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760768.012890:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760768.012894:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760768.012896:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760768.012936:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760768.012945:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760768.012970:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760768.014250:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac000 - object 0x240000400:5 offset 26955776<>26959872 00000100:00100000:3.0:1789760768.014321:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512704896:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760768.055816:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973900 gen 3 00000008:00010000:3.0:1789760768.055853:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:3.0:1789760768.055860:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760768.055868:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760768.055887:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:3.0:1789760768.055945:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512704896:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760768.056027:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6581, 6581] 00000080:00200000:2.0:1789760768.056039:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760768.056044:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760768.056056:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26959872 i_blocks 53255 00000020:00200000:2.0:1789760768.056059:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760768.056067:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760768.056093:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26959872, count: 4096 00000080:00200000:2.0:1789760768.056099:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26959872, bytes: 4096 00000080:00200000:2.0:1789760768.056106:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760768.056113:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760768.056115:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26959872, 26963968) 0 0 00000080:00200000:2.0:1789760768.056120:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760768.056135:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6582, 6582] 00020000:00200000:2.0:1789760768.056142:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26959872 chunk: [26959872, 26963968) 26963968, 4096 00020000:00200000:2.0:1789760768.056146:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760768.056163:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6582, 6582] 00000008:00010000:2.0:1789760768.056454:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760768.056460:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760768.056471:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26959872, 26963968) append 0 append lockpos 0 00000080:00200000:2.0:1789760768.056483:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26959872=0x19b6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760768.056530:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760768.056533:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26963968 00000020:00010000:2.0:1789760768.056539:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760768.056543:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.056545:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.056549:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760768.056551:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760768.056558:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760768.056563:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760768.056578:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760768.058747:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 26959872<>26963968 00000100:00100000:3.0:1789760768.059186:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760768.059207:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512705024:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760768.119740:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973901 gen 3 00000008:00010000:3.0:1789760768.119776:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760768.119784:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760768.119792:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760768.119849:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760768.119874:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512705024:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760768.122026:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6582, 6582] 00000080:00200000:2.0:1789760768.122044:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760768.122049:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760768.122061:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26963968 i_blocks 53255 00000020:00200000:2.0:1789760768.122065:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760768.122076:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760768.122102:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26963968, count: 4096 00000080:00200000:2.0:1789760768.122108:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26963968, bytes: 4096 00000080:00200000:2.0:1789760768.122119:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760768.122126:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760768.122128:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26963968, 26968064) 0 0 00000080:00200000:2.0:1789760768.122132:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760768.122144:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6583, 6583] 00020000:00200000:2.0:1789760768.122151:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26963968 chunk: [26963968, 26968064) 26968064, 4096 00020000:00200000:2.0:1789760768.122154:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760768.122170:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6583, 6583] 00000008:00010000:2.0:1789760768.122250:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760768.122256:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760768.122264:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26963968, 26968064) append 0 append lockpos 0 00000080:00200000:2.0:1789760768.122275:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26963968=0x19b7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760768.122325:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760768.122328:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26968064 00000020:00010000:2.0:1789760768.122335:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760768.122338:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.122341:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.122345:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760768.122347:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760768.122355:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760768.122360:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760768.122383:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760768.122459:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 26963968<>26968064 00000100:00100000:3.0:1789760768.122535:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512705152:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760768.163467:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973902 gen 3 00000008:00010000:3.0:1789760768.163505:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760768.163511:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760768.163525:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760768.163615:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760768.163641:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512705152:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760768.168209:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6583, 6583] 00000080:00200000:2.0:1789760768.168242:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760768.168248:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760768.168262:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26968064 i_blocks 53255 00000020:00200000:2.0:1789760768.168266:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760768.168279:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760768.168304:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26968064, count: 4096 00000080:00200000:2.0:1789760768.168311:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26968064, bytes: 4096 00000080:00200000:2.0:1789760768.168321:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760768.168328:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760768.168330:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26968064, 26972160) 0 0 00000080:00200000:2.0:1789760768.168335:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760768.168348:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6584, 6584] 00020000:00200000:2.0:1789760768.168354:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26968064 chunk: [26968064, 26972160) 26972160, 4096 00020000:00200000:2.0:1789760768.168358:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760768.168375:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6584, 6584] 00000008:00010000:2.0:1789760768.168389:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760768.168395:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760768.168404:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26968064, 26972160) append 0 append lockpos 0 00000080:00200000:2.0:1789760768.168416:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26968064=0x19b8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760768.168467:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760768.168469:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26972160 00000020:00010000:2.0:1789760768.168476:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760768.168479:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.168481:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760768.168484:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760768.168487:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760768.168493:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760768.168499:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760768.168520:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760768.168598:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 26968064<>26972160 00000100:00100000:3.0:1789760768.168684:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512705280:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760768.232374:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973903 gen 3 00000008:00010000:3.0:1789760768.232411:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760768.232418:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760768.232427:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760768.232449:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760768.232472:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512705280:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760768.232574:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6584, 6584] 00000080:00200000:0.0:1789760768.232587:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.232591:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.232603:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26972160 i_blocks 53255 00000020:00200000:0.0:1789760768.232605:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.232679:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.232707:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26972160, count: 4096 00000080:00200000:0.0:1789760768.232713:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26972160, bytes: 4096 00000080:00200000:0.0:1789760768.232721:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.232728:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.232730:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26972160, 26976256) 0 0 00000080:00200000:0.0:1789760768.232733:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.232747:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6585, 6585] 00020000:00200000:0.0:1789760768.232755:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26972160 chunk: [26972160, 26976256) 26976256, 4096 00020000:00200000:0.0:1789760768.232759:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.232774:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6585, 6585] 00000008:00010000:0.0:1789760768.232857:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.232862:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760768.233161:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26972160, 26976256) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.233173:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26972160=0x19b9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.233217:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.233219:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26976256 00000020:00010000:0.0:1789760768.233238:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.233242:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.233244:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.233248:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.233250:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.233257:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.233263:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.233278:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc480 00000008:00100000:0.0:1789760768.233353:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149c00 - object 0x240000400:5 offset 26972160<>26976256 00000100:00100000:1.0:1789760768.233434:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512705408:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760768.297323:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973904 gen 3 00000008:00010000:1.0:1789760768.297363:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760768.297370:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.297380:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.297402:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760768.297427:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512705408:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760768.297593:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6585, 6585] 00000080:00200000:0.0:1789760768.297606:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.297611:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.297623:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26976256 i_blocks 53255 00000020:00200000:0.0:1789760768.297627:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.297634:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.297659:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26976256, count: 4096 00000080:00200000:0.0:1789760768.297664:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26976256, bytes: 4096 00000080:00200000:0.0:1789760768.297672:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.297679:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.297681:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26976256, 26980352) 0 0 00000080:00200000:0.0:1789760768.297685:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.297696:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6586, 6586] 00020000:00200000:0.0:1789760768.297702:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26976256 chunk: [26976256, 26980352) 26980352, 4096 00020000:00200000:0.0:1789760768.297706:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.297719:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6586, 6586] 00000008:00010000:0.0:1789760768.297733:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.297738:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760768.297747:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26976256, 26980352) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.297757:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26976256=0x19ba000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.297800:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.297802:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26980352 00000020:00010000:0.0:1789760768.297809:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.297813:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.297815:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.297818:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.297821:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.297826:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.297831:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.297844:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc840 00000008:00100000:0.0:1789760768.298083:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149f80 - object 0x240000400:5 offset 26976256<>26980352 00000100:00100000:1.0:1789760768.298125:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760768.298182:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512705536:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760768.340877:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760768.340890:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.340931:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.341003:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760768.341028:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512705536:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760768.341086:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6586, 6586] 00000080:00200000:0.0:1789760768.341099:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.341104:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.341116:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26980352 i_blocks 53255 00000020:00200000:0.0:1789760768.341120:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.341127:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.341151:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26980352, count: 4096 00000080:00200000:0.0:1789760768.341157:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26980352, bytes: 4096 00000080:00200000:0.0:1789760768.341163:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.341170:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.341171:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26980352, 26984448) 0 0 00000080:00200000:0.0:1789760768.341175:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.341187:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6587, 6587] 00020000:00200000:0.0:1789760768.341193:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26980352 chunk: [26980352, 26984448) 26984448, 4096 00020000:00200000:0.0:1789760768.341197:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.341212:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6587, 6587] 00000008:00010000:0.0:1789760768.341234:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.341239:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760768.341249:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26980352, 26984448) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.341259:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26980352=0x19bb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.341302:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.341339:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26984448 00000020:00010000:0.0:1789760768.341347:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.341351:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.341354:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.341358:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.341361:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.341367:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.341373:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.341387:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc840 00000008:00100000:0.0:1789760768.341466:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148000 - object 0x240000400:5 offset 26980352<>26984448 00000100:00100000:0.0:1789760768.341512:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760768.341526:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512705664:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760768.395017:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973906 gen 3 00000008:00010000:0.0:1789760768.395061:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:0.0:1789760768.395068:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760768.395076:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760768.395147:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:0.0:1789760768.395171:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512705664:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760768.401174:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6587, 6587] 00000080:00200000:1.0:1789760768.401195:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760768.401200:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760768.401215:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26984448 i_blocks 53255 00000020:00200000:1.0:1789760768.401220:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760768.401245:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760768.401273:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26984448, count: 4096 00000080:00200000:1.0:1789760768.401280:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26984448, bytes: 4096 00000080:00200000:1.0:1789760768.401293:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760768.401300:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760768.401302:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26984448, 26988544) 0 0 00000080:00200000:1.0:1789760768.401306:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760768.401320:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6588, 6588] 00020000:00200000:1.0:1789760768.401327:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26984448 chunk: [26984448, 26988544) 26988544, 4096 00020000:00200000:1.0:1789760768.401331:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760768.401348:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6588, 6588] 00000008:00010000:1.0:1789760768.401362:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760768.401366:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760768.401376:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26984448, 26988544) append 0 append lockpos 0 00000080:00200000:1.0:1789760768.401388:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26984448=0x19bc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760768.401444:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760768.401447:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26988544 00000020:00010000:1.0:1789760768.401453:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760768.401457:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.401460:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.401463:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760768.401465:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760768.401472:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760768.401478:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760768.401501:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760768.401579:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 26984448<>26988544 00000100:00100000:0.0:1789760768.401670:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512705792:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760768.457099:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973907 gen 3 00000008:00010000:0.0:1789760768.457136:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760768.457143:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760768.457150:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760768.457172:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760768.457194:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512705792:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760768.457571:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6588, 6588] 00000080:00200000:1.0:1789760768.457584:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760768.457590:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760768.457618:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26988544 i_blocks 53255 00000020:00200000:1.0:1789760768.457622:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760768.457631:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760768.457658:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26988544, count: 4096 00000080:00200000:1.0:1789760768.457663:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26988544, bytes: 4096 00000080:00200000:1.0:1789760768.457672:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760768.457680:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760768.457682:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26988544, 26992640) 0 0 00000080:00200000:1.0:1789760768.457686:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760768.457699:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6589, 6589] 00020000:00200000:1.0:1789760768.457706:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26988544 chunk: [26988544, 26992640) 26992640, 4096 00020000:00200000:1.0:1789760768.457709:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760768.457723:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6589, 6589] 00000008:00010000:1.0:1789760768.457736:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760768.457740:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760768.457750:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26988544, 26992640) append 0 append lockpos 0 00000080:00200000:1.0:1789760768.457761:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26988544=0x19bd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760768.457818:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760768.457821:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26992640 00000020:00010000:1.0:1789760768.457827:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760768.457831:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.457833:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.457836:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760768.457839:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760768.457844:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760768.457850:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760768.457864:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760768.457979:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94700 - object 0x240000400:5 offset 26988544<>26992640 00000100:00100000:0.0:1789760768.458124:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512705920:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760768.499405:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973908 gen 3 00000008:00010000:0.0:1789760768.499441:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760768.499447:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760768.499455:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760768.499475:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760768.499497:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512705920:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760768.499875:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6589, 6589] 00000080:00200000:1.0:1789760768.499886:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760768.499891:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760768.499938:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26992640 i_blocks 53255 00000020:00200000:1.0:1789760768.499943:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760768.499951:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760768.499976:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26992640, count: 4096 00000080:00200000:1.0:1789760768.499983:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26992640, bytes: 4096 00000080:00200000:1.0:1789760768.499991:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760768.499998:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760768.500000:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26992640, 26996736) 0 0 00000080:00200000:1.0:1789760768.500004:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760768.500017:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6590, 6590] 00020000:00200000:1.0:1789760768.500023:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26992640 chunk: [26992640, 26996736) 26996736, 4096 00020000:00200000:1.0:1789760768.500026:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760768.500041:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6590, 6590] 00000008:00010000:1.0:1789760768.500054:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760768.500058:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760768.500066:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26992640, 26996736) append 0 append lockpos 0 00000080:00200000:1.0:1789760768.500076:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26992640=0x19be000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760768.500115:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760768.500118:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 26996736 00000020:00010000:1.0:1789760768.500124:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760768.500127:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.500130:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.500133:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760768.500136:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760768.500143:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760768.500148:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760768.500161:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760768.500247:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 26992640<>26996736 00000100:00100000:0.0:1789760768.500383:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760768.500397:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512706048:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760768.535605:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973909 gen 3 00000008:00010000:0.0:1789760768.535643:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760768.535650:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760768.535657:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760768.535678:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760768.535700:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512706048:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760768.535806:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6590, 6590] 00000080:00200000:1.0:1789760768.535819:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760768.535824:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760768.535837:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 26996736 i_blocks 53255 00000020:00200000:1.0:1789760768.535840:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760768.535849:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760768.536046:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 26996736, count: 4096 00000080:00200000:1.0:1789760768.536053:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 26996736, bytes: 4096 00000080:00200000:1.0:1789760768.536062:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760768.536069:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760768.536071:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [26996736, 27000832) 0 0 00000080:00200000:1.0:1789760768.536074:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760768.536087:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6591, 6591] 00020000:00200000:1.0:1789760768.536094:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 26996736 chunk: [26996736, 27000832) 27000832, 4096 00020000:00200000:1.0:1789760768.536097:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760768.536112:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6591, 6591] 00000008:00010000:1.0:1789760768.536125:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760768.536130:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760768.536140:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [26996736, 27000832) append 0 append lockpos 0 00000080:00200000:1.0:1789760768.536151:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=26996736=0x19bf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760768.536193:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760768.536197:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27000832 00000020:00010000:1.0:1789760768.536203:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760768.536206:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.536209:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.536212:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760768.536214:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760768.536232:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760768.536238:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760768.536251:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760768.536326:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 26996736<>27000832 00000100:00100000:0.0:1789760768.536686:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512706176:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760768.586470:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973910 gen 3 00000008:00010000:0.0:1789760768.586571:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760768.586578:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760768.586587:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760768.586654:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760768.586678:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512706176:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760768.586752:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6591, 6591] 00000080:00200000:1.0:1789760768.586765:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760768.586770:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760768.586848:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27000832 i_blocks 53255 00000020:00200000:1.0:1789760768.586853:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760768.586861:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760768.586887:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27000832, count: 4096 00000080:00200000:1.0:1789760768.586894:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27000832, bytes: 4096 00000080:00200000:1.0:1789760768.586924:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760768.586932:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760768.586934:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27000832, 27004928) 0 0 00000080:00200000:1.0:1789760768.586938:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760768.586951:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6592, 6592] 00020000:00200000:1.0:1789760768.586958:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27000832 chunk: [27000832, 27004928) 27004928, 4096 00020000:00200000:1.0:1789760768.586961:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760768.586976:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6592, 6592] 00000008:00010000:1.0:1789760768.586989:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760768.586994:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760768.587002:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27000832, 27004928) append 0 append lockpos 0 00000080:00200000:1.0:1789760768.587013:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27000832=0x19c0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760768.587056:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760768.587059:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27004928 00000020:00010000:1.0:1789760768.587065:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760768.587069:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.587071:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.587074:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760768.587077:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760768.587083:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760768.587089:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760768.587102:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760768.587176:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95c00 - object 0x240000400:5 offset 27000832<>27004928 00000100:00100000:0.0:1789760768.587244:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512706304:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760768.671338:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760768.671349:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760768.671357:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760768.671378:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760768.671402:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512706304:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760768.671533:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6592, 6592] 00000080:00200000:1.0:1789760768.671548:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760768.671553:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760768.671565:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27004928 i_blocks 53255 00000020:00200000:1.0:1789760768.671569:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760768.671577:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760768.671607:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27004928, count: 4096 00000080:00200000:1.0:1789760768.671614:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27004928, bytes: 4096 00000080:00200000:1.0:1789760768.671622:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760768.671630:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760768.671632:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27004928, 27009024) 0 0 00000080:00200000:1.0:1789760768.671636:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760768.671650:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6593, 6593] 00020000:00200000:1.0:1789760768.671656:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27004928 chunk: [27004928, 27009024) 27009024, 4096 00020000:00200000:1.0:1789760768.671660:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760768.671675:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6593, 6593] 00000008:00010000:1.0:1789760768.671688:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760768.671693:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760768.671701:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27004928, 27009024) append 0 append lockpos 0 00000080:00200000:1.0:1789760768.671712:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27004928=0x19c1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760768.671757:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760768.671759:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27009024 00000020:00010000:1.0:1789760768.671766:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760768.672470:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.672485:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.672489:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760768.672491:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760768.672499:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760768.672505:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760768.672523:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760768.672599:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 27004928<>27009024 00000100:00100000:0.0:1789760768.672745:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512706432:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760768.710485:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973911 gen 3 00000100:00100000:0.0:1789760768.710515:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005c97800 x1876699512706432/t4294973912(4294973912) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760784 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760768.710551:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760768.710556:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760768.710563:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760768.710585:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760768.710605:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512706432:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760768.710692:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6593, 6593] 00000080:00200000:1.0:1789760768.710705:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760768.710710:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760768.710721:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27009024 i_blocks 53255 00000020:00200000:1.0:1789760768.710725:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760768.710732:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760768.710758:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27009024, count: 4096 00000080:00200000:1.0:1789760768.710765:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27009024, bytes: 4096 00000080:00200000:1.0:1789760768.710773:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760768.710923:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760768.710925:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27009024, 27013120) 0 0 00000080:00200000:1.0:1789760768.710929:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760768.710942:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6594, 6594] 00020000:00200000:1.0:1789760768.710949:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27009024 chunk: [27009024, 27013120) 27013120, 4096 00020000:00200000:1.0:1789760768.710953:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760768.710968:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6594, 6594] 00000008:00010000:1.0:1789760768.710981:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760768.710986:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760768.710995:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27009024, 27013120) append 0 append lockpos 0 00000080:00200000:1.0:1789760768.711005:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27009024=0x19c2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760768.711047:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760768.711050:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27013120 00000020:00010000:1.0:1789760768.711058:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760768.711061:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.711063:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.711066:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760768.711068:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760768.711074:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760768.711079:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760768.711092:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760768.711168:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 27009024<>27013120 00000100:00100000:0.0:1789760768.711297:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512706560:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760768.760576:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973913 gen 3 00000008:00010000:0.0:1789760768.760621:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760768.760628:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760768.760635:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760768.760657:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760768.760679:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512706560:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760768.760770:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6594, 6594] 00000080:00200000:1.0:1789760768.760782:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760768.760787:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760768.760798:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27013120 i_blocks 53255 00000020:00200000:1.0:1789760768.760802:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760768.760809:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760768.760834:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27013120, count: 4096 00000080:00200000:1.0:1789760768.760840:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27013120, bytes: 4096 00000080:00200000:1.0:1789760768.760847:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760768.760854:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760768.760856:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27013120, 27017216) 0 0 00000080:00200000:1.0:1789760768.760859:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760768.760871:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6595, 6595] 00020000:00200000:1.0:1789760768.760878:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27013120 chunk: [27013120, 27017216) 27017216, 4096 00020000:00200000:1.0:1789760768.760881:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760768.760895:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6595, 6595] 00000008:00010000:1.0:1789760768.760933:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760768.760938:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760768.760947:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27013120, 27017216) append 0 append lockpos 0 00000080:00200000:1.0:1789760768.760957:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27013120=0x19c3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760768.762108:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760768.762114:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27017216 00000020:00010000:1.0:1789760768.762124:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760768.762127:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.762131:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.762134:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760768.762136:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760768.762144:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760768.762152:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760768.762172:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760768.762262:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97100 - object 0x240000400:5 offset 27013120<>27017216 00000100:00100000:0.0:1789760768.762342:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512706688:192.168.203.156@tcp:4:dd.0 00010000:00010000:0.0:1789760768.798279:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760768.811475:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973914 gen 3 00000008:00010000:0.0:1789760768.811517:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760768.811524:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760768.811533:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760768.811632:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760768.811656:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512706688:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760768.811701:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6595, 6595] 00000080:00200000:3.0:1789760768.811714:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760768.811720:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760768.811731:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27017216 i_blocks 53255 00000020:00200000:3.0:1789760768.811735:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760768.811742:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760768.811767:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27017216, count: 4096 00000080:00200000:3.0:1789760768.811773:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27017216, bytes: 4096 00000080:00200000:3.0:1789760768.811781:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760768.811788:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760768.811790:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27017216, 27021312) 0 0 00000080:00200000:3.0:1789760768.811794:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760768.811809:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6596, 6596] 00020000:00200000:3.0:1789760768.811815:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27017216 chunk: [27017216, 27021312) 27021312, 4096 00020000:00200000:3.0:1789760768.811819:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760768.811834:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6596, 6596] 00000008:00010000:3.0:1789760768.811847:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760768.811851:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760768.811861:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27017216, 27021312) append 0 append lockpos 0 00000080:00200000:3.0:1789760768.811871:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27017216=0x19c4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760768.811938:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760768.811941:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27021312 00000020:00010000:3.0:1789760768.811947:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760768.811950:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760768.811954:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760768.811957:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760768.811960:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760768.811966:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760768.811986:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760768.812000:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760768.813057:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac000 - object 0x240000400:5 offset 27017216<>27021312 00000100:00100000:3.0:1789760768.813102:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760768.813119:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512706816:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760768.880346:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973915 gen 3 00000008:00010000:3.0:1789760768.880386:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:3.0:1789760768.880392:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760768.880403:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760768.880429:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:3.0:1789760768.880453:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512706816:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760768.880554:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6596, 6596] 00000080:00200000:1.0:1789760768.880565:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760768.880570:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760768.880582:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27021312 i_blocks 53255 00000020:00200000:1.0:1789760768.880585:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760768.880592:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760768.880628:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27021312, count: 4096 00000080:00200000:1.0:1789760768.880634:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27021312, bytes: 4096 00000080:00200000:1.0:1789760768.880642:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760768.880650:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760768.880653:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27021312, 27025408) 0 0 00000080:00200000:1.0:1789760768.880656:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760768.880671:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6597, 6597] 00020000:00200000:1.0:1789760768.880679:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27021312 chunk: [27021312, 27025408) 27025408, 4096 00020000:00200000:1.0:1789760768.880682:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760768.880698:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6597, 6597] 00000008:00010000:1.0:1789760768.880712:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760768.880717:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760768.880726:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27021312, 27025408) append 0 append lockpos 0 00000080:00200000:1.0:1789760768.880735:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27021312=0x19c5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760768.880779:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760768.880782:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27025408 00000020:00010000:1.0:1789760768.880788:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760768.880790:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.880793:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760768.880796:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760768.880799:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760768.880807:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760768.880812:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760768.880828:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760768.883069:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 27021312<>27025408 00000100:00100000:1.0:1789760768.883426:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512706944:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760768.932134:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973916 gen 3 00000008:00010000:1.0:1789760768.932169:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760768.932175:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760768.932184:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760768.932205:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760768.932237:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512706944:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760768.932365:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6597, 6597] 00000080:00200000:0.0:1789760768.932377:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760768.932382:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760768.932396:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27025408 i_blocks 53255 00000020:00200000:0.0:1789760768.932399:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760768.932408:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760768.932434:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27025408, count: 4096 00000080:00200000:0.0:1789760768.932439:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27025408, bytes: 4096 00000080:00200000:0.0:1789760768.932451:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760768.932458:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760768.932460:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27025408, 27029504) 0 0 00000080:00200000:0.0:1789760768.932464:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760768.932478:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6598, 6598] 00020000:00200000:0.0:1789760768.932485:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27025408 chunk: [27025408, 27029504) 27029504, 4096 00020000:00200000:0.0:1789760768.932488:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760768.932503:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6598, 6598] 00000008:00010000:0.0:1789760768.932517:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760768.932522:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760768.932530:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27025408, 27029504) append 0 append lockpos 0 00000080:00200000:0.0:1789760768.932541:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27025408=0x19c6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760768.932583:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760768.932586:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27029504 00000020:00010000:0.0:1789760768.932593:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760768.932596:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.932600:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760768.932603:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760768.932605:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760768.932611:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760768.932618:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760768.932632:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc840 00000008:00100000:0.0:1789760768.932705:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148000 - object 0x240000400:5 offset 27025408<>27029504 00000100:00100000:1.0:1789760768.932800:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512707072:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760769.003966:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973917 gen 3 00000008:00010000:1.0:1789760769.004003:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760769.004010:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.004021:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.004044:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760769.004067:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512707072:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760769.004247:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6598, 6598] 00000080:00200000:0.0:1789760769.004259:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.004264:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.004276:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27029504 i_blocks 53255 00000020:00200000:0.0:1789760769.004280:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.004288:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.004316:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27029504, count: 4096 00000080:00200000:0.0:1789760769.004322:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27029504, bytes: 4096 00000080:00200000:0.0:1789760769.004331:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.004339:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.004341:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27029504, 27033600) 0 0 00000080:00200000:0.0:1789760769.004345:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.004359:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6599, 6599] 00020000:00200000:0.0:1789760769.004366:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27029504 chunk: [27029504, 27033600) 27033600, 4096 00020000:00200000:0.0:1789760769.004369:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.004384:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6599, 6599] 00000008:00010000:0.0:1789760769.004397:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.004402:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760769.004410:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27029504, 27033600) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.004422:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27029504=0x19c7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.004464:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.004466:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27033600 00000020:00010000:0.0:1789760769.004473:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.004476:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.004479:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.004482:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.004484:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.004491:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.004496:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.004509:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc480 00000008:00100000:0.0:1789760769.004593:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149f80 - object 0x240000400:5 offset 27029504<>27033600 00000100:00100000:0.0:1789760769.004638:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760769.004650:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512707200:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760769.043712:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973918 gen 3 00000008:00010000:0.0:1789760769.043750:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:0.0:1789760769.043757:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760769.043768:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760769.043838:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:0.0:1789760769.043863:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512707200:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760769.044439:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6599, 6599] 00000080:00200000:3.0:1789760769.044452:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760769.044457:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760769.044468:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27033600 i_blocks 53255 00000020:00200000:3.0:1789760769.044472:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760769.044480:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760769.044505:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27033600, count: 4096 00000080:00200000:3.0:1789760769.044511:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27033600, bytes: 4096 00000080:00200000:3.0:1789760769.044519:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760769.044527:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760769.044529:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27033600, 27037696) 0 0 00000080:00200000:3.0:1789760769.044533:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760769.044608:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6600, 6600] 00020000:00200000:3.0:1789760769.044615:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27033600 chunk: [27033600, 27037696) 27037696, 4096 00020000:00200000:3.0:1789760769.044619:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760769.044634:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6600, 6600] 00000008:00010000:3.0:1789760769.044648:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760769.044652:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760769.044661:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27033600, 27037696) append 0 append lockpos 0 00000080:00200000:3.0:1789760769.044671:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27033600=0x19c8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760769.044779:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760769.044782:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27037696 00000020:00010000:3.0:1789760769.045022:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760769.045025:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760769.045084:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760769.045089:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760769.045092:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760769.045099:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760769.045106:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760769.045125:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760769.045204:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac000 - object 0x240000400:5 offset 27033600<>27037696 00000100:00100000:2.0:1789760769.045277:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512707328:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760769.089389:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973919 gen 3 00000008:00010000:3.0:1789760769.089428:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:3.0:1789760769.089435:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760769.089446:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760769.089468:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:3.0:1789760769.089493:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512707328:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760769.089707:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6600, 6600] 00000080:00200000:0.0:1789760769.089720:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.089726:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.089738:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27037696 i_blocks 53255 00000020:00200000:0.0:1789760769.089741:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.089749:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.089775:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27037696, count: 4096 00000080:00200000:0.0:1789760769.089781:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27037696, bytes: 4096 00000080:00200000:0.0:1789760769.089838:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.089846:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.089848:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27037696, 27041792) 0 0 00000080:00200000:0.0:1789760769.089852:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.089865:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6601, 6601] 00020000:00200000:0.0:1789760769.089872:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27037696 chunk: [27037696, 27041792) 27041792, 4096 00020000:00200000:0.0:1789760769.089876:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.089992:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6601, 6601] 00000008:00010000:0.0:1789760769.090007:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.090013:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760769.090022:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27037696, 27041792) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.090032:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27037696=0x19c9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.090075:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.090078:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27041792 00000020:00010000:0.0:1789760769.090084:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.090087:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.090090:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.090093:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.090096:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.090102:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.090107:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.090116:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc480 00000008:00100000:0.0:1789760769.090284:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149f80 - object 0x240000400:5 offset 27037696<>27041792 00000100:00100000:1.0:1789760769.090363:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512707456:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760769.135300:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760769.135310:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.135320:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.135375:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760769.135399:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512707456:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760769.135451:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6601, 6601] 00000080:00200000:3.0:1789760769.135463:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760769.135469:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760769.135479:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27041792 i_blocks 53255 00000020:00200000:3.0:1789760769.135482:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760769.135490:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760769.135515:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27041792, count: 4096 00000080:00200000:3.0:1789760769.135520:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27041792, bytes: 4096 00000080:00200000:3.0:1789760769.135528:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760769.135535:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760769.135537:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27041792, 27045888) 0 0 00000080:00200000:3.0:1789760769.135541:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760769.135554:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6602, 6602] 00020000:00200000:3.0:1789760769.135560:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27041792 chunk: [27041792, 27045888) 27045888, 4096 00020000:00200000:3.0:1789760769.135563:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760769.135578:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6602, 6602] 00000008:00010000:3.0:1789760769.135590:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760769.135594:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760769.135603:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27041792, 27045888) append 0 append lockpos 0 00000080:00200000:3.0:1789760769.135614:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27041792=0x19ca000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760769.135659:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760769.135661:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27045888 00000020:00010000:3.0:1789760769.135668:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760769.135671:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760769.135674:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760769.135677:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760769.135679:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760769.135685:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760769.135690:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760769.135704:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760769.135784:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac000 - object 0x240000400:5 offset 27041792<>27045888 00000100:00100000:2.0:1789760769.135839:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512707584:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760769.164535:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973920 gen 3 00000100:00100000:3.0:1789760769.164564:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc0107ac000 x1876699512707584/t4294973921(4294973921) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760785 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760769.164601:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:3.0:1789760769.164607:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760769.164617:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760769.164640:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:3.0:1789760769.164660:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512707584:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760769.164741:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6602, 6602] 00000080:00200000:0.0:1789760769.164753:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.164758:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.164769:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27045888 i_blocks 53255 00000020:00200000:0.0:1789760769.164772:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.164780:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.164805:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27045888, count: 4096 00000080:00200000:0.0:1789760769.164811:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27045888, bytes: 4096 00000080:00200000:0.0:1789760769.164818:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.164825:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.164827:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27045888, 27049984) 0 0 00000080:00200000:0.0:1789760769.164831:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.164843:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6603, 6603] 00020000:00200000:0.0:1789760769.164850:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27045888 chunk: [27045888, 27049984) 27049984, 4096 00020000:00200000:0.0:1789760769.164854:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.164866:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6603, 6603] 00000008:00010000:0.0:1789760769.164880:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.164885:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760769.164925:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27045888, 27049984) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.164935:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27045888=0x19cb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.164976:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.164978:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27049984 00000020:00010000:0.0:1789760769.164984:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.164987:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.164990:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.164993:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.164995:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.165001:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.165007:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.165019:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc240 00000008:00100000:0.0:1789760769.165122:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149500 - object 0x240000400:5 offset 27045888<>27049984 00000100:00100000:1.0:1789760769.165363:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760769.165378:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512707712:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760769.190817:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973922 gen 3 00000008:00010000:1.0:1789760769.190997:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760769.191002:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.191009:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.191064:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760769.191083:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512707712:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760769.191140:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6603, 6603] 00000080:00200000:0.0:1789760769.191154:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.191158:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.191170:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27049984 i_blocks 53255 00000020:00200000:0.0:1789760769.191173:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.191181:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.191355:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27049984, count: 4096 00000080:00200000:0.0:1789760769.191363:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27049984, bytes: 4096 00000080:00200000:0.0:1789760769.191372:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.191378:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.191379:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27049984, 27054080) 0 0 00000080:00200000:0.0:1789760769.191383:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.191396:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6604, 6604] 00020000:00200000:0.0:1789760769.191401:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27049984 chunk: [27049984, 27054080) 27054080, 4096 00020000:00200000:0.0:1789760769.191404:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.191416:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6604, 6604] 00000008:00010000:0.0:1789760769.191425:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.191428:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760769.191436:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27049984, 27054080) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.191445:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27049984=0x19cc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.191477:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.191479:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27054080 00000020:00010000:0.0:1789760769.191484:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.191486:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.191488:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.191490:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.191492:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.191496:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.191500:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.191511:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc300 00000008:00100000:0.0:1789760769.191817:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514bb80 - object 0x240000400:5 offset 27049984<>27054080 00000100:00100000:1.0:1789760769.191878:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512707840:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760769.229147:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973923 gen 3 00000008:00010000:1.0:1789760769.229184:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760769.229190:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.229199:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.229432:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760769.229457:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512707840:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760769.229525:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6604, 6604] 00000080:00200000:0.0:1789760769.229537:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.229542:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.229551:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27054080 i_blocks 53255 00000020:00200000:0.0:1789760769.229554:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.229561:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.229587:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27054080, count: 4096 00000080:00200000:0.0:1789760769.229593:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27054080, bytes: 4096 00000080:00200000:0.0:1789760769.229600:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.229608:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.229609:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27054080, 27058176) 0 0 00000080:00200000:0.0:1789760769.229679:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.229693:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6605, 6605] 00020000:00200000:0.0:1789760769.229701:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27054080 chunk: [27054080, 27058176) 27058176, 4096 00020000:00200000:0.0:1789760769.229705:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.229721:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6605, 6605] 00000008:00010000:0.0:1789760769.229735:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.229740:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760769.229749:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27054080, 27058176) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.229759:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27054080=0x19cd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.229813:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.229815:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27058176 00000020:00010000:0.0:1789760769.229822:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.229825:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.229827:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.229830:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.229833:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.229840:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.229845:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.229859:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccb40 00000008:00100000:0.0:1789760769.230157:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b100 - object 0x240000400:5 offset 27054080<>27058176 00000100:00100000:1.0:1789760769.230203:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760769.230780:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512707968:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760769.272660:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973924 gen 3 00000008:00010000:1.0:1789760769.272700:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760769.272707:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.272717:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.272738:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760769.272762:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512707968:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760769.272921:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6605, 6605] 00000080:00200000:3.0:1789760769.272935:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760769.272941:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760769.272952:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27058176 i_blocks 53255 00000020:00200000:3.0:1789760769.272955:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760769.272962:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760769.272988:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27058176, count: 4096 00000080:00200000:3.0:1789760769.272995:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27058176, bytes: 4096 00000080:00200000:3.0:1789760769.273003:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760769.273010:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760769.273012:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27058176, 27062272) 0 0 00000080:00200000:3.0:1789760769.273015:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760769.273029:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6606, 6606] 00020000:00200000:3.0:1789760769.273036:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27058176 chunk: [27058176, 27062272) 27062272, 4096 00020000:00200000:3.0:1789760769.273040:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760769.273055:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6606, 6606] 00000008:00010000:3.0:1789760769.273068:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760769.273072:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760769.273081:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27058176, 27062272) append 0 append lockpos 0 00000080:00200000:3.0:1789760769.273091:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27058176=0x19ce000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760769.273126:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760769.273128:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27062272 00000020:00010000:3.0:1789760769.273134:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760769.273137:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760769.273140:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760769.273142:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760769.273144:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760769.273150:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760769.273155:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760769.273237:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:1.0:1789760769.273333:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 27058176<>27062272 00000100:00100000:1.0:1789760769.273374:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512708096:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760769.326281:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:1.0:1789760769.326292:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.326636:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.326707:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:1.0:1789760769.326731:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512708096:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760769.326840:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6606, 6606] 00000080:00200000:3.0:1789760769.326852:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760769.326857:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760769.326868:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27062272 i_blocks 53255 00000020:00200000:3.0:1789760769.326872:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760769.326879:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760769.326934:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27062272, count: 4096 00000080:00200000:3.0:1789760769.326941:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27062272, bytes: 4096 00000080:00200000:3.0:1789760769.326950:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760769.326958:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760769.326961:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27062272, 27066368) 0 0 00000080:00200000:3.0:1789760769.326965:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760769.326979:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6607, 6607] 00020000:00200000:3.0:1789760769.326985:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27062272 chunk: [27062272, 27066368) 27066368, 4096 00020000:00200000:3.0:1789760769.326988:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760769.327004:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6607, 6607] 00000008:00010000:3.0:1789760769.327018:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760769.327022:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760769.327032:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27062272, 27066368) append 0 append lockpos 0 00000080:00200000:3.0:1789760769.327043:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27062272=0x19cf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760769.327145:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760769.327149:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27066368 00000020:00010000:3.0:1789760769.327156:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760769.327159:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760769.327162:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760769.327165:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760769.327169:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760769.327176:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760769.327182:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760769.327197:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760769.327289:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 27062272<>27066368 00000100:00100000:3.0:1789760769.328083:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760769.328103:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512708224:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760769.384867:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973926 gen 3 00000008:00010000:3.0:1789760769.384998:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760769.385005:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760769.385014:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760769.385036:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760769.385057:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512708224:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760769.385096:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6607, 6607] 00000080:00200000:1.0:1789760769.385109:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760769.385114:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760769.385131:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27066368 i_blocks 53255 00000020:00200000:1.0:1789760769.385135:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760769.385143:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760769.385174:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27066368, count: 4096 00000080:00200000:1.0:1789760769.385180:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27066368, bytes: 4096 00000080:00200000:1.0:1789760769.385188:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760769.385196:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760769.385198:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27066368, 27070464) 0 0 00000080:00200000:1.0:1789760769.385202:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760769.385224:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6608, 6608] 00020000:00200000:1.0:1789760769.385230:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27066368 chunk: [27066368, 27070464) 27070464, 4096 00020000:00200000:1.0:1789760769.385233:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760769.385247:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6608, 6608] 00000008:00010000:1.0:1789760769.385270:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760769.385275:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760769.385284:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27066368, 27070464) append 0 append lockpos 0 00000080:00200000:1.0:1789760769.385303:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27066368=0x19d0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760769.385478:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760769.385481:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27070464 00000020:00010000:1.0:1789760769.385487:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760769.385490:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.385492:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.385495:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760769.385498:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760769.385504:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760769.385510:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760769.385523:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760769.385597:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 27066368<>27070464 00000100:00100000:0.0:1789760769.385675:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512708352:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760769.448181:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973927 gen 3 00000008:00010000:0.0:1789760769.448229:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760769.448236:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760769.448244:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760769.448263:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760769.448285:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512708352:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760769.448512:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6608, 6608] 00000080:00200000:1.0:1789760769.448525:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760769.448530:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760769.448542:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27070464 i_blocks 53255 00000020:00200000:1.0:1789760769.448545:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760769.448553:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760769.448580:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27070464, count: 4096 00000080:00200000:1.0:1789760769.448586:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27070464, bytes: 4096 00000080:00200000:1.0:1789760769.448595:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760769.448602:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760769.448604:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27070464, 27074560) 0 0 00000080:00200000:1.0:1789760769.448669:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760769.448683:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6609, 6609] 00020000:00200000:1.0:1789760769.448690:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27070464 chunk: [27070464, 27074560) 27074560, 4096 00020000:00200000:1.0:1789760769.448693:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760769.448708:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6609, 6609] 00000008:00010000:1.0:1789760769.448722:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760769.448726:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760769.448736:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27070464, 27074560) append 0 append lockpos 0 00000080:00200000:1.0:1789760769.448747:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27070464=0x19d1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760769.448841:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760769.448844:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27074560 00000020:00010000:1.0:1789760769.448851:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760769.448854:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.448857:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.448860:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760769.448863:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760769.448870:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760769.448877:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760769.448996:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760769.449073:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97100 - object 0x240000400:5 offset 27070464<>27074560 00000100:00100000:0.0:1789760769.449165:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512708480:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760769.484132:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973928 gen 3 00000008:00010000:0.0:1789760769.484169:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760769.484176:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760769.484184:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760769.484203:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760769.484235:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512708480:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760769.484403:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6609, 6609] 00000080:00200000:1.0:1789760769.484417:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760769.484423:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760769.484438:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27074560 i_blocks 53255 00000020:00200000:1.0:1789760769.484441:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760769.484449:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760769.484482:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27074560, count: 4096 00000080:00200000:1.0:1789760769.484488:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27074560, bytes: 4096 00000080:00200000:1.0:1789760769.484495:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760769.484502:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760769.484504:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27074560, 27078656) 0 0 00000080:00200000:1.0:1789760769.484508:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760769.484521:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6610, 6610] 00020000:00200000:1.0:1789760769.484527:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27074560 chunk: [27074560, 27078656) 27078656, 4096 00020000:00200000:1.0:1789760769.484531:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760769.484545:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6610, 6610] 00000008:00010000:1.0:1789760769.484557:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760769.484562:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760769.484570:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27074560, 27078656) append 0 append lockpos 0 00000080:00200000:1.0:1789760769.484581:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27074560=0x19d2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760769.484695:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760769.484698:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27078656 00000020:00010000:1.0:1789760769.484705:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760769.484708:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.484711:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.484714:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760769.484717:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760769.484722:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760769.484729:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760769.484742:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760769.485157:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 27074560<>27078656 00000100:00100000:0.0:1789760769.485222:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512708608:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760769.526427:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973929 gen 3 00000008:00010000:0.0:1789760769.526464:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760769.526470:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760769.526479:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760769.526499:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760769.526524:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512708608:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760769.526619:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6610, 6610] 00000080:00200000:3.0:1789760769.526632:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760769.526636:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760769.526648:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27078656 i_blocks 53255 00000020:00200000:3.0:1789760769.526651:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760769.526659:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760769.526692:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27078656, count: 4096 00000080:00200000:3.0:1789760769.526699:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27078656, bytes: 4096 00000080:00200000:3.0:1789760769.526707:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760769.526714:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760769.526716:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27078656, 27082752) 0 0 00000080:00200000:3.0:1789760769.526720:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760769.526735:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6611, 6611] 00020000:00200000:3.0:1789760769.526743:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27078656 chunk: [27078656, 27082752) 27082752, 4096 00020000:00200000:3.0:1789760769.526747:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760769.526763:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6611, 6611] 00000008:00010000:3.0:1789760769.526779:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760769.526785:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760769.526795:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27078656, 27082752) append 0 append lockpos 0 00000080:00200000:3.0:1789760769.526807:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27078656=0x19d3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760769.526853:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760769.526856:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27082752 00000020:00010000:3.0:1789760769.526863:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760769.526867:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760769.526870:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760769.526873:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760769.526876:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760769.526894:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760769.526933:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760769.526948:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760769.527024:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 27078656<>27082752 00000100:00100000:2.0:1789760769.527177:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512708736:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760769.573196:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973930 gen 3 00000008:00010000:3.0:1789760769.573237:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760769.573244:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760769.573253:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760769.573346:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760769.573370:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512708736:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760769.573717:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6611, 6611] 00000080:00200000:0.0:1789760769.573730:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.573735:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.573746:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27082752 i_blocks 53255 00000020:00200000:0.0:1789760769.573750:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.573758:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.573782:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27082752, count: 4096 00000080:00200000:0.0:1789760769.573788:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27082752, bytes: 4096 00000080:00200000:0.0:1789760769.573796:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.573802:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.573804:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27082752, 27086848) 0 0 00000080:00200000:0.0:1789760769.573808:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.573822:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6612, 6612] 00020000:00200000:0.0:1789760769.573828:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27082752 chunk: [27082752, 27086848) 27086848, 4096 00020000:00200000:0.0:1789760769.573832:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.573847:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6612, 6612] 00000008:00010000:0.0:1789760769.573860:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.573864:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760769.573873:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27082752, 27086848) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.573884:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27082752=0x19d4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.573973:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.573976:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27086848 00000020:00010000:0.0:1789760769.573983:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.573986:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.573989:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.573992:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.573995:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.574001:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.574006:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.574020:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000008:00100000:0.0:1789760769.574098:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b100 - object 0x240000400:5 offset 27082752<>27086848 00000100:00100000:1.0:1789760769.574363:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512708864:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760769.616663:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760769.616671:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.616678:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.616698:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760769.616723:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512708864:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760769.616895:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6612, 6612] 00000080:00200000:3.0:1789760769.616922:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760769.616926:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760769.616937:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27086848 i_blocks 53255 00000020:00200000:3.0:1789760769.616939:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760769.616946:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760769.616966:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27086848, count: 4096 00000080:00200000:3.0:1789760769.616971:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27086848, bytes: 4096 00000080:00200000:3.0:1789760769.616977:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760769.616983:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760769.616984:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27086848, 27090944) 0 0 00000080:00200000:3.0:1789760769.616987:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760769.616997:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6613, 6613] 00020000:00200000:3.0:1789760769.617002:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27086848 chunk: [27086848, 27090944) 27090944, 4096 00020000:00200000:3.0:1789760769.617005:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760769.617016:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6613, 6613] 00000008:00010000:3.0:1789760769.617027:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760769.617031:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760769.617037:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27086848, 27090944) append 0 append lockpos 0 00000080:00200000:3.0:1789760769.617045:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27086848=0x19d5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760769.617077:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760769.617079:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27090944 00000020:00010000:3.0:1789760769.617084:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760769.617086:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760769.617088:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760769.617091:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760769.617093:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760769.617097:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760769.617100:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760769.617111:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760769.617187:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 27086848<>27090944 00000100:00100000:3.0:1789760769.617234:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760769.617247:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512708992:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760769.656814:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973932 gen 3 00000008:00010000:3.0:1789760769.656862:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760769.656868:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760769.656877:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760769.656898:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760769.656947:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512708992:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760769.657012:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6613, 6613] 00000080:00200000:0.0:1789760769.657026:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.657031:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.657042:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27090944 i_blocks 53255 00000020:00200000:0.0:1789760769.657046:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.657054:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.657080:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27090944, count: 4096 00000080:00200000:0.0:1789760769.657086:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27090944, bytes: 4096 00000080:00200000:0.0:1789760769.657093:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.657100:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.657102:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27090944, 27095040) 0 0 00000080:00200000:0.0:1789760769.657105:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.657116:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6614, 6614] 00020000:00200000:0.0:1789760769.657122:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27090944 chunk: [27090944, 27095040) 27095040, 4096 00020000:00200000:0.0:1789760769.657126:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.657140:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6614, 6614] 00000008:00010000:0.0:1789760769.657153:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.657157:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760769.657166:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27090944, 27095040) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.657177:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27090944=0x19d6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.657225:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.657227:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27095040 00000020:00010000:0.0:1789760769.657236:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.657239:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.657242:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.657245:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.657248:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.657253:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.657259:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760769.657277:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc900 00000008:00100000:3.0:1789760769.657351:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 27090944<>27095040 00000100:00100000:3.0:1789760769.657385:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512709120:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760769.704967:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973933 gen 3 00000008:00010000:3.0:1789760769.705009:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:3.0:1789760769.705016:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760769.705025:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760769.705045:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:3.0:1789760769.705073:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512709120:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760769.705442:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6614, 6614] 00000080:00200000:0.0:1789760769.705455:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.705460:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.705474:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27095040 i_blocks 53255 00000020:00200000:0.0:1789760769.705478:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.705487:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.705516:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27095040, count: 4096 00000080:00200000:0.0:1789760769.705522:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27095040, bytes: 4096 00000080:00200000:0.0:1789760769.705666:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.705677:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.705680:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27095040, 27099136) 0 0 00000080:00200000:0.0:1789760769.705684:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.705703:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6615, 6615] 00020000:00200000:0.0:1789760769.705710:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27095040 chunk: [27095040, 27099136) 27099136, 4096 00020000:00200000:0.0:1789760769.705714:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.705733:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6615, 6615] 00000008:00010000:0.0:1789760769.705749:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.705755:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760769.705766:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27095040, 27099136) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.705779:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27095040=0x19d7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.706143:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.706147:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27099136 00000020:00010000:0.0:1789760769.706157:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.706160:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.706162:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.706311:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.706314:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.706322:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.706329:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.706348:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cccc0 00000008:00100000:0.0:1789760769.706447:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514bb80 - object 0x240000400:5 offset 27095040<>27099136 00000100:00100000:1.0:1789760769.706719:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512709248:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760769.776394:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973934 gen 3 00000008:00010000:1.0:1789760769.776429:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760769.776435:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.776443:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.776621:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760769.776645:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512709248:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760769.776853:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6615, 6615] 00000080:00200000:0.0:1789760769.776865:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.776870:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.777221:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27099136 i_blocks 53255 00000020:00200000:0.0:1789760769.777226:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.777234:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.777260:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27099136, count: 4096 00000080:00200000:0.0:1789760769.777266:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27099136, bytes: 4096 00000080:00200000:0.0:1789760769.777274:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.777281:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.777283:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27099136, 27103232) 0 0 00000080:00200000:0.0:1789760769.777287:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.777300:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6616, 6616] 00020000:00200000:0.0:1789760769.777307:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27099136 chunk: [27099136, 27103232) 27103232, 4096 00020000:00200000:0.0:1789760769.777311:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.777326:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6616, 6616] 00000008:00010000:0.0:1789760769.777339:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.777344:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760769.777353:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27099136, 27103232) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.777363:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27099136=0x19d8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.777406:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.777409:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27103232 00000020:00010000:0.0:1789760769.777416:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.777419:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.777423:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.777426:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.777429:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.777436:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.777442:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.777457:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000008:00100000:0.0:1789760769.777539:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149500 - object 0x240000400:5 offset 27099136<>27103232 00000100:00100000:1.0:1789760769.777604:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512709376:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760769.821474:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973935 gen 3 00000008:00010000:1.0:1789760769.821520:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760769.821526:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.821536:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.821552:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760769.821577:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512709376:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760769.821620:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512709632:192.168.203.156@tcp:400:kworker.0 00000100:00100000:1.0:1789760769.821704:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1789760769.821725:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e6cf7267-9b19-462a-a263-14115df15dab:2358:1876699512709504:192.168.203.156@tcp:400:kworker.0 00000080:00200000:0.0:1789760769.821728:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6616, 6616] 00000080:00200000:0.0:1789760769.821742:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.821747:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.821759:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27103232 i_blocks 53255 00000100:00100000:1.0:1789760769.821762:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2358:1876699512709760:192.168.203.156@tcp:400:kworker.0 00000020:00200000:0.0:1789760769.821765:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.821775:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.821801:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27103232, count: 4096 00000080:00200000:0.0:1789760769.821807:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27103232, bytes: 4096 00000080:00200000:0.0:1789760769.821815:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.821821:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.821823:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27103232, 27107328) 0 0 00000080:00200000:0.0:1789760769.821827:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.821839:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6617, 6617] 00020000:00200000:0.0:1789760769.821846:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27103232 chunk: [27103232, 27107328) 27107328, 4096 00020000:00200000:0.0:1789760769.821850:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.821863:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6617, 6617] 00000008:00010000:0.0:1789760769.821874:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.821879:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760769.822176:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27103232, 27107328) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.822190:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27103232=0x19d9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.822242:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.822245:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27107328 00000020:00010000:0.0:1789760769.822251:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.822255:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.822258:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.822262:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.822265:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.822272:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.822277:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.822289:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc600 00000008:00100000:0.0:1789760769.822355:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149180 - object 0x240000400:5 offset 27103232<>27107328 00000100:00100000:0.0:1789760769.822387:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760769.822397:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512709888:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760769.822507:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760769.825649:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e6cf7267-9b19-462a-a263-14115df15dab:2358:1876699512709504:192.168.203.156@tcp:400:kworker.0 00000100:00100000:1.0:1789760769.825669:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2358:1876699512709760:192.168.203.156@tcp:400:kworker.0 00000100:00100000:1.0:1789760769.825680:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512709632:192.168.203.156@tcp:400:kworker.0 00000008:00010000:0.0:1789760769.868061:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:0.0:1789760769.868073:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760769.868085:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760769.868109:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:0.0:1789760769.868133:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512709888:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760769.868238:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6617, 6617] 00000080:00200000:1.0:1789760769.868250:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760769.868256:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760769.868267:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27107328 i_blocks 53255 00000020:00200000:1.0:1789760769.868271:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760769.868286:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760769.868313:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27107328, count: 4096 00000080:00200000:1.0:1789760769.868319:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27107328, bytes: 4096 00000080:00200000:1.0:1789760769.868327:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760769.868335:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760769.868337:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27107328, 27111424) 0 0 00000080:00200000:1.0:1789760769.868341:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760769.868355:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6618, 6618] 00020000:00200000:1.0:1789760769.868362:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27107328 chunk: [27107328, 27111424) 27111424, 4096 00020000:00200000:1.0:1789760769.868366:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760769.868392:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6618, 6618] 00000008:00010000:1.0:1789760769.868407:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760769.868412:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760769.868421:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27107328, 27111424) append 0 append lockpos 0 00000080:00200000:1.0:1789760769.868432:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27107328=0x19da000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760769.868475:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760769.868478:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27111424 00000020:00010000:1.0:1789760769.868484:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760769.868488:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.868491:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.868494:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760769.868497:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760769.868504:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760769.868510:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760769.868523:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760769.868597:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 27107328<>27111424 00000100:00100000:1.0:1789760769.868644:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760769.868668:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512710016:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760769.910064:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973937 gen 3 00000008:00010000:1.0:1789760769.910110:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760769.910118:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.910126:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.910150:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760769.910174:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512710016:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760769.910352:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6618, 6618] 00000080:00200000:0.0:1789760769.910363:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.910368:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.910378:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27111424 i_blocks 53255 00000020:00200000:0.0:1789760769.910381:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.910388:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.910414:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27111424, count: 4096 00000080:00200000:0.0:1789760769.910420:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27111424, bytes: 4096 00000080:00200000:0.0:1789760769.910427:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.910433:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.910435:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27111424, 27115520) 0 0 00000080:00200000:0.0:1789760769.910438:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.910450:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6619, 6619] 00020000:00200000:0.0:1789760769.910457:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27111424 chunk: [27111424, 27115520) 27115520, 4096 00020000:00200000:0.0:1789760769.910461:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.910474:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6619, 6619] 00000008:00010000:0.0:1789760769.910487:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.910491:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760769.910501:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27111424, 27115520) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.910511:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27111424=0x19db000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.910553:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.910556:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27115520 00000020:00010000:0.0:1789760769.910563:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.910566:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.910569:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.910572:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.910574:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.910581:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.910585:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.910598:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc600 00000008:00100000:0.0:1789760769.910686:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b800 - object 0x240000400:5 offset 27111424<>27115520 00000100:00100000:0.0:1789760769.910720:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760769.910732:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512710144:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760769.944376:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973938 gen 3 00000008:00010000:0.0:1789760769.944410:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:0.0:1789760769.944416:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760769.944425:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760769.944447:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:0.0:1789760769.944472:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512710144:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760769.944572:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6619, 6619] 00000080:00200000:1.0:1789760769.944585:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760769.944590:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760769.944613:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27115520 i_blocks 53255 00000020:00200000:1.0:1789760769.944617:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760769.944625:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760769.944655:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27115520, count: 4096 00000080:00200000:1.0:1789760769.944661:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27115520, bytes: 4096 00000080:00200000:1.0:1789760769.944671:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760769.944678:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760769.944680:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27115520, 27119616) 0 0 00000080:00200000:1.0:1789760769.944684:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760769.944698:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6620, 6620] 00020000:00200000:1.0:1789760769.944705:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27115520 chunk: [27115520, 27119616) 27119616, 4096 00020000:00200000:1.0:1789760769.944709:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760769.944725:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6620, 6620] 00000008:00010000:1.0:1789760769.944738:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760769.944744:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760769.944752:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27115520, 27119616) append 0 append lockpos 0 00000080:00200000:1.0:1789760769.944763:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27115520=0x19dc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760769.944817:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760769.944820:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27119616 00000020:00010000:1.0:1789760769.944827:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760769.944830:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.944833:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760769.944836:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760769.944839:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760769.944845:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760769.944851:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760769.944864:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760769.945200:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 27115520<>27119616 00000100:00100000:1.0:1789760769.945242:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760769.945256:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512710272:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760769.984804:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760769.984814:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760769.984823:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760769.984840:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760769.984863:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512710272:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760769.984975:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6620, 6620] 00000080:00200000:0.0:1789760769.984989:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760769.984994:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760769.985006:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27119616 i_blocks 53255 00000020:00200000:0.0:1789760769.985010:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760769.985018:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760769.985041:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27119616, count: 4096 00000080:00200000:0.0:1789760769.985047:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27119616, bytes: 4096 00000080:00200000:0.0:1789760769.985054:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760769.985061:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760769.985063:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27119616, 27123712) 0 0 00000080:00200000:0.0:1789760769.985067:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760769.985080:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6621, 6621] 00020000:00200000:0.0:1789760769.985086:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27119616 chunk: [27119616, 27123712) 27123712, 4096 00020000:00200000:0.0:1789760769.985090:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760769.985104:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6621, 6621] 00000008:00010000:0.0:1789760769.985117:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760769.985121:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760769.985131:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27119616, 27123712) append 0 append lockpos 0 00000080:00200000:0.0:1789760769.985141:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27119616=0x19dd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760769.985183:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760769.985186:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27123712 00000020:00010000:0.0:1789760769.985193:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760769.985196:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.985199:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760769.985202:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760769.985205:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760769.985221:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760769.985228:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760769.985241:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc600 00000008:00100000:0.0:1789760769.985311:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b800 - object 0x240000400:5 offset 27119616<>27123712 00000100:00100000:0.0:1789760769.985345:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760769.985357:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512710400:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760770.020114:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973940 gen 3 00000008:00010000:0.0:1789760770.020159:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:0.0:1789760770.020166:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760770.020175:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760770.020242:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:0.0:1789760770.020264:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512710400:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760770.020330:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6621, 6621] 00000080:00200000:2.0:1789760770.020342:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760770.020347:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760770.020359:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27123712 i_blocks 53255 00000020:00200000:2.0:1789760770.020362:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760770.020370:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760770.020395:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27123712, count: 4096 00000080:00200000:2.0:1789760770.020402:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27123712, bytes: 4096 00000080:00200000:2.0:1789760770.020409:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760770.020416:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760770.020418:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27123712, 27127808) 0 0 00000080:00200000:2.0:1789760770.020422:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760770.020434:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6622, 6622] 00020000:00200000:2.0:1789760770.020442:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27123712 chunk: [27123712, 27127808) 27127808, 4096 00020000:00200000:2.0:1789760770.020445:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760770.020459:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6622, 6622] 00000008:00010000:2.0:1789760770.020472:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760770.020477:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760770.020485:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27123712, 27127808) append 0 append lockpos 0 00000080:00200000:2.0:1789760770.020496:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27123712=0x19de000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760770.020539:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760770.020541:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27127808 00000020:00010000:2.0:1789760770.020547:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760770.020550:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.020553:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.020556:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760770.020558:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760770.020565:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760770.020570:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760770.020583:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760770.020716:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 27123712<>27127808 00000100:00100000:3.0:1789760770.020812:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760770.020827:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512710528:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760770.056945:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973941 gen 3 00000008:00010000:3.0:1789760770.056984:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760770.056991:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.057000:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760770.057020:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760770.057043:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512710528:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760770.057159:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6622, 6622] 00000080:00200000:2.0:1789760770.057171:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760770.057176:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760770.057187:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27127808 i_blocks 53255 00000020:00200000:2.0:1789760770.057199:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760770.057207:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760770.057240:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27127808, count: 4096 00000080:00200000:2.0:1789760770.057246:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27127808, bytes: 4096 00000080:00200000:2.0:1789760770.057253:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760770.057261:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760770.057263:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27127808, 27131904) 0 0 00000080:00200000:2.0:1789760770.057267:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760770.057280:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6623, 6623] 00020000:00200000:2.0:1789760770.057286:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27127808 chunk: [27127808, 27131904) 27131904, 4096 00020000:00200000:2.0:1789760770.057290:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760770.057304:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6623, 6623] 00000008:00010000:2.0:1789760770.057316:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760770.057320:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760770.057328:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27127808, 27131904) append 0 append lockpos 0 00000080:00200000:2.0:1789760770.057339:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27127808=0x19df000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760770.057378:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760770.057381:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27131904 00000020:00010000:2.0:1789760770.057387:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760770.057390:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.057393:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.057403:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760770.057405:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760770.057412:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760770.057417:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760770.057430:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760770.057500:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 27127808<>27131904 00000100:00100000:3.0:1789760770.057586:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512710656:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760770.114720:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973942 gen 3 00000008:00010000:3.0:1789760770.114759:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760770.114765:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.114962:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760770.114980:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760770.115005:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512710656:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760770.115475:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6623, 6623] 00000080:00200000:2.0:1789760770.115489:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760770.115493:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760770.115506:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27131904 i_blocks 53255 00000020:00200000:2.0:1789760770.115509:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760770.115518:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760770.115545:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27131904, count: 4096 00000080:00200000:2.0:1789760770.115551:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27131904, bytes: 4096 00000080:00200000:2.0:1789760770.115561:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760770.115569:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760770.115571:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27131904, 27136000) 0 0 00000080:00200000:2.0:1789760770.115574:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760770.115589:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6624, 6624] 00020000:00200000:2.0:1789760770.115596:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27131904 chunk: [27131904, 27136000) 27136000, 4096 00020000:00200000:2.0:1789760770.115600:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760770.115614:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6624, 6624] 00000008:00010000:2.0:1789760770.115627:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760770.115632:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760770.115641:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27131904, 27136000) append 0 append lockpos 0 00000080:00200000:2.0:1789760770.115652:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27131904=0x19e0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760770.115696:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760770.115699:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27136000 00000020:00010000:2.0:1789760770.115706:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760770.115709:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.115712:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.115715:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760770.115718:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760770.115724:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760770.115730:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760770.116457:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760770.116553:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 27131904<>27136000 00000100:00100000:3.0:1789760770.116663:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760770.116681:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512710784:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760770.153587:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973943 gen 3 00000008:00010000:3.0:1789760770.153695:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760770.153702:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.153710:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760770.153729:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760770.153752:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512710784:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760770.154085:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6624, 6624] 00000080:00200000:2.0:1789760770.154097:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760770.154101:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760770.154112:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27136000 i_blocks 53255 00000020:00200000:2.0:1789760770.154117:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760770.154125:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760770.154151:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27136000, count: 4096 00000080:00200000:2.0:1789760770.154157:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27136000, bytes: 4096 00000080:00200000:2.0:1789760770.154165:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760770.154172:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760770.154174:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27136000, 27140096) 0 0 00000080:00200000:2.0:1789760770.154178:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760770.154192:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6625, 6625] 00020000:00200000:2.0:1789760770.154198:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27136000 chunk: [27136000, 27140096) 27140096, 4096 00020000:00200000:2.0:1789760770.154202:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760770.154229:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6625, 6625] 00000008:00010000:2.0:1789760770.154313:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760770.154318:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760770.154328:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27136000, 27140096) append 0 append lockpos 0 00000080:00200000:2.0:1789760770.154339:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27136000=0x19e1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760770.154383:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760770.154386:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27140096 00000020:00010000:2.0:1789760770.154393:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760770.154397:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.154399:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760770.154403:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760770.154405:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760770.154413:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760770.154419:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760770.154434:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760770.154509:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 27136000<>27140096 00000100:00100000:2.0:1789760770.154541:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760770.154553:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512710912:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760770.199002:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973944 gen 3 00000008:00010000:3.0:1789760770.199036:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760770.199042:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.199052:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760770.199071:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760770.199093:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512710912:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760770.199251:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6625, 6625] 00000080:00200000:0.0:1789760770.199263:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760770.199268:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760770.199278:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27140096 i_blocks 53255 00000020:00200000:0.0:1789760770.199282:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760770.199289:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760770.199311:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27140096, count: 4096 00000080:00200000:0.0:1789760770.199316:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27140096, bytes: 4096 00000080:00200000:0.0:1789760770.199323:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760770.199330:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760770.199332:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27140096, 27144192) 0 0 00000080:00200000:0.0:1789760770.199335:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760770.199346:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6626, 6626] 00020000:00200000:0.0:1789760770.199353:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27140096 chunk: [27140096, 27144192) 27144192, 4096 00020000:00200000:0.0:1789760770.199356:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760770.199369:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6626, 6626] 00000008:00010000:0.0:1789760770.199380:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760770.199385:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760770.199394:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27140096, 27144192) append 0 append lockpos 0 00000080:00200000:0.0:1789760770.199403:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27140096=0x19e2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760770.199441:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760770.199443:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27144192 00000020:00010000:0.0:1789760770.199450:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760770.199453:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.199456:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.199460:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760770.199462:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760770.199468:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760770.199473:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760770.199485:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc600 00000008:00100000:0.0:1789760770.199553:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b800 - object 0x240000400:5 offset 27140096<>27144192 00000100:00100000:1.0:1789760770.199643:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512711040:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760770.234014:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973945 gen 3 00000008:00010000:1.0:1789760770.234046:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760770.234052:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760770.234059:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760770.234078:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760770.234098:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512711040:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760770.234189:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6626, 6626] 00000080:00200000:0.0:1789760770.234200:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760770.234205:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760770.234225:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27144192 i_blocks 53255 00000020:00200000:0.0:1789760770.234228:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760770.234297:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760770.234321:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27144192, count: 4096 00000080:00200000:0.0:1789760770.234327:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27144192, bytes: 4096 00000080:00200000:0.0:1789760770.234334:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760770.234340:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760770.234342:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27144192, 27148288) 0 0 00000080:00200000:0.0:1789760770.234346:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760770.234357:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6627, 6627] 00020000:00200000:0.0:1789760770.234364:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27144192 chunk: [27144192, 27148288) 27148288, 4096 00020000:00200000:0.0:1789760770.234367:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760770.234381:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6627, 6627] 00000008:00010000:0.0:1789760770.234393:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760770.234397:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760770.234405:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27144192, 27148288) append 0 append lockpos 0 00000080:00200000:0.0:1789760770.234415:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27144192=0x19e3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760770.234454:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760770.234456:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27148288 00000020:00010000:0.0:1789760770.234462:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760770.234465:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.234468:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.234472:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760770.234475:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760770.234481:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760770.234486:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760770.234498:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc540 00000008:00100000:0.0:1789760770.234567:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148380 - object 0x240000400:5 offset 27144192<>27148288 00000100:00100000:1.0:1789760770.234626:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512711168:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760770.263801:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973946 gen 3 00000008:00010000:1.0:1789760770.263835:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760770.263841:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760770.263851:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760770.263869:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760770.263890:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512711168:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760770.264005:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6627, 6627] 00000080:00200000:0.0:1789760770.264016:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760770.264021:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760770.264033:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27148288 i_blocks 53255 00000020:00200000:0.0:1789760770.264036:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760770.264044:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760770.264069:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27148288, count: 4096 00000080:00200000:0.0:1789760770.264074:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27148288, bytes: 4096 00000080:00200000:0.0:1789760770.264080:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760770.264088:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760770.264090:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27148288, 27152384) 0 0 00000080:00200000:0.0:1789760770.264093:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760770.264106:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6628, 6628] 00020000:00200000:0.0:1789760770.264112:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27148288 chunk: [27148288, 27152384) 27152384, 4096 00020000:00200000:0.0:1789760770.264116:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760770.264128:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6628, 6628] 00000008:00010000:0.0:1789760770.264141:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760770.264146:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760770.264155:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27148288, 27152384) append 0 append lockpos 0 00000080:00200000:0.0:1789760770.264165:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27148288=0x19e4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760770.264221:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760770.264224:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27152384 00000020:00010000:0.0:1789760770.264325:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760770.264329:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.264332:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.264336:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760770.264339:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760770.264345:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760770.264351:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760770.264364:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cca80 00000008:00100000:0.0:1789760770.264438:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b480 - object 0x240000400:5 offset 27148288<>27152384 00000100:00100000:1.0:1789760770.264523:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512711296:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760770.304951:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973947 gen 3 00000008:00010000:1.0:1789760770.304991:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760770.304997:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760770.305006:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760770.305024:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760770.305045:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512711296:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760770.305139:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6628, 6628] 00000080:00200000:0.0:1789760770.305154:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760770.305160:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760770.305172:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27152384 i_blocks 53255 00000020:00200000:0.0:1789760770.305176:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760770.305183:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760770.305218:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27152384, count: 4096 00000080:00200000:0.0:1789760770.305225:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27152384, bytes: 4096 00000080:00200000:0.0:1789760770.305293:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760770.305300:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760770.305302:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27152384, 27156480) 0 0 00000080:00200000:0.0:1789760770.305306:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760770.305318:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6629, 6629] 00020000:00200000:0.0:1789760770.305325:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27152384 chunk: [27152384, 27156480) 27156480, 4096 00020000:00200000:0.0:1789760770.305329:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760770.305343:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6629, 6629] 00000008:00010000:0.0:1789760770.305356:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760770.305360:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760770.305369:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27152384, 27156480) append 0 append lockpos 0 00000080:00200000:0.0:1789760770.305381:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27152384=0x19e5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760770.305428:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760770.305431:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27156480 00000020:00010000:0.0:1789760770.305438:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760770.305441:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.305444:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.305448:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760770.305450:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760770.305456:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760770.305461:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760770.305474:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccf00 00000008:00100000:0.0:1789760770.305547:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149880 - object 0x240000400:5 offset 27152384<>27156480 00000100:00100000:1.0:1789760770.305716:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760770.305731:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512711424:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760770.348081:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973948 gen 3 00000008:00010000:1.0:1789760770.348119:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760770.348126:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760770.348136:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760770.348154:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760770.348186:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512711424:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760770.348336:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6629, 6629] 00000080:00200000:0.0:1789760770.348349:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760770.348354:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760770.348367:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27156480 i_blocks 53255 00000020:00200000:0.0:1789760770.348371:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760770.348378:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760770.348403:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27156480, count: 4096 00000080:00200000:0.0:1789760770.348408:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27156480, bytes: 4096 00000080:00200000:0.0:1789760770.348415:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760770.348422:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760770.348424:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27156480, 27160576) 0 0 00000080:00200000:0.0:1789760770.348428:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760770.348440:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6630, 6630] 00020000:00200000:0.0:1789760770.348448:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27156480 chunk: [27156480, 27160576) 27160576, 4096 00020000:00200000:0.0:1789760770.348451:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760770.348466:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6630, 6630] 00000008:00010000:0.0:1789760770.348477:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760770.348481:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760770.348490:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27156480, 27160576) append 0 append lockpos 0 00000080:00200000:0.0:1789760770.348500:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27156480=0x19e6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760770.348541:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760770.348543:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27160576 00000020:00010000:0.0:1789760770.348549:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760770.348552:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.348555:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.348559:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760770.348561:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760770.348567:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760770.348573:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760770.348585:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc000 00000008:00100000:0.0:1789760770.348666:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a300 - object 0x240000400:5 offset 27156480<>27160576 00000100:00100000:1.0:1789760770.349072:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512711552:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760770.385854:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973949 gen 3 00000008:00010000:1.0:1789760770.385889:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760770.385895:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760770.385924:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760770.385943:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760770.385963:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512711552:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760770.386082:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6630, 6630] 00000080:00200000:0.0:1789760770.386094:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760770.386099:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760770.386109:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27160576 i_blocks 53255 00000020:00200000:0.0:1789760770.386112:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760770.386120:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760770.386143:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27160576, count: 4096 00000080:00200000:0.0:1789760770.386149:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27160576, bytes: 4096 00000080:00200000:0.0:1789760770.386156:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760770.386163:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760770.386165:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27160576, 27164672) 0 0 00000080:00200000:0.0:1789760770.386169:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760770.386181:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6631, 6631] 00020000:00200000:0.0:1789760770.386188:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27160576 chunk: [27160576, 27164672) 27164672, 4096 00020000:00200000:0.0:1789760770.386191:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760770.386219:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6631, 6631] 00000008:00010000:0.0:1789760770.386242:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760770.386247:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760770.386256:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27160576, 27164672) append 0 append lockpos 0 00000080:00200000:0.0:1789760770.386267:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27160576=0x19e7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760770.386310:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760770.386313:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27164672 00000020:00010000:0.0:1789760770.386320:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760770.386323:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.386326:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.386330:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760770.386332:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760770.386338:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760770.386343:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760770.386357:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000008:00100000:0.0:1789760770.386434:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a680 - object 0x240000400:5 offset 27160576<>27164672 00000100:00100000:1.0:1789760770.386631:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512711680:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760770.418826:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760770.418837:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760770.418846:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760770.418864:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760770.418894:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512711680:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760770.419131:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6631, 6631] 00000080:00200000:0.0:1789760770.419145:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760770.419150:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760770.419161:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27164672 i_blocks 53255 00000020:00200000:0.0:1789760770.419164:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760770.419171:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760770.419195:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27164672, count: 4096 00000080:00200000:0.0:1789760770.419201:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27164672, bytes: 4096 00000080:00200000:0.0:1789760770.419218:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760770.419225:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760770.419227:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27164672, 27168768) 0 0 00000080:00200000:0.0:1789760770.419290:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760770.419302:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6632, 6632] 00020000:00200000:0.0:1789760770.419309:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27164672 chunk: [27164672, 27168768) 27168768, 4096 00020000:00200000:0.0:1789760770.419313:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760770.419328:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6632, 6632] 00000008:00010000:0.0:1789760770.419341:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760770.419345:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760770.419355:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27164672, 27168768) append 0 append lockpos 0 00000080:00200000:0.0:1789760770.419365:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27164672=0x19e8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760770.419410:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760770.419413:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27168768 00000020:00010000:0.0:1789760770.419419:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760770.419422:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.419425:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.419429:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760770.419432:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760770.419438:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760770.419444:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760770.419458:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc780 00000008:00100000:0.0:1789760770.419541:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148e00 - object 0x240000400:5 offset 27164672<>27168768 00000100:00100000:1.0:1789760770.419869:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512711808:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760770.457220:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973951 gen 3 00000008:00010000:1.0:1789760770.457275:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760770.457282:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760770.457290:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760770.457312:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760770.457336:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512711808:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760770.457699:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6632, 6632] 00000080:00200000:0.0:1789760770.457713:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760770.457718:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760770.457731:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27168768 i_blocks 53255 00000020:00200000:0.0:1789760770.457734:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760770.457742:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760770.457770:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27168768, count: 4096 00000080:00200000:0.0:1789760770.457777:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27168768, bytes: 4096 00000080:00200000:0.0:1789760770.457784:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760770.457803:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760770.457806:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27168768, 27172864) 0 0 00000080:00200000:0.0:1789760770.457810:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760770.457823:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6633, 6633] 00020000:00200000:0.0:1789760770.457830:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27168768 chunk: [27168768, 27172864) 27172864, 4096 00020000:00200000:0.0:1789760770.457833:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760770.457849:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6633, 6633] 00000008:00010000:0.0:1789760770.457862:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760770.457867:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760770.457877:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27168768, 27172864) append 0 append lockpos 0 00000080:00200000:0.0:1789760770.457888:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27168768=0x19e9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760770.457969:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760770.457972:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27172864 00000020:00010000:0.0:1789760770.457979:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760770.457982:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.457985:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.457989:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760770.457991:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760770.457998:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760770.458005:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760770.458021:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccd80 00000008:00100000:0.0:1789760770.458101:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148a80 - object 0x240000400:5 offset 27168768<>27172864 00000100:00100000:1.0:1789760770.458204:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512711936:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760770.495199:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760770.495221:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760770.495301:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760770.495320:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760770.495344:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512711936:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760770.495731:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6633, 6633] 00000080:00200000:0.0:1789760770.495743:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760770.495748:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760770.495759:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27172864 i_blocks 53255 00000020:00200000:0.0:1789760770.495763:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760770.495771:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760770.495808:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27172864, count: 4096 00000080:00200000:0.0:1789760770.495814:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27172864, bytes: 4096 00000080:00200000:0.0:1789760770.495822:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760770.495829:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760770.495831:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27172864, 27176960) 0 0 00000080:00200000:0.0:1789760770.495835:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760770.495848:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6634, 6634] 00020000:00200000:0.0:1789760770.495856:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27172864 chunk: [27172864, 27176960) 27176960, 4096 00020000:00200000:0.0:1789760770.495859:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760770.495883:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6634, 6634] 00000008:00010000:0.0:1789760770.495896:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760770.495927:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760770.495937:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27172864, 27176960) append 0 append lockpos 0 00000080:00200000:0.0:1789760770.495947:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27172864=0x19ea000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760770.495990:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760770.495992:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27176960 00000020:00010000:0.0:1789760770.495999:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760770.496003:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.496005:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.496008:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760770.496011:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760770.496018:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760770.496024:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760770.496038:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc180 00000008:00100000:0.0:1789760770.496113:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514ad80 - object 0x240000400:5 offset 27172864<>27176960 00000100:00100000:1.0:1789760770.496325:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512712064:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760770.535441:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973952 gen 3 00000100:00100000:1.0:1789760770.535474:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00514ad80 x1876699512712064/t4294973953(4294973953) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760786 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760770.535512:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760770.535518:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760770.535527:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760770.535549:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760770.535570:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512712064:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760770.535988:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6634, 6634] 00000080:00200000:0.0:1789760770.536002:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760770.536008:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760770.536021:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27176960 i_blocks 53255 00000020:00200000:0.0:1789760770.536024:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760770.536034:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760770.536065:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27176960, count: 4096 00000080:00200000:0.0:1789760770.536072:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27176960, bytes: 4096 00000080:00200000:0.0:1789760770.536081:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760770.536088:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760770.536090:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27176960, 27181056) 0 0 00000080:00200000:0.0:1789760770.536094:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760770.536107:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6635, 6635] 00020000:00200000:0.0:1789760770.536115:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27176960 chunk: [27176960, 27181056) 27181056, 4096 00020000:00200000:0.0:1789760770.536118:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760770.536134:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6635, 6635] 00000008:00010000:0.0:1789760770.536149:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760770.536154:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760770.536162:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27176960, 27181056) append 0 append lockpos 0 00000080:00200000:0.0:1789760770.536173:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27176960=0x19eb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760770.536469:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760770.536473:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27181056 00000020:00010000:0.0:1789760770.536481:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760770.536484:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.536487:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760770.536491:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760770.536494:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760770.536501:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760770.536507:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760770.536523:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000008:00100000:0.0:1789760770.536678:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148700 - object 0x240000400:5 offset 27176960<>27181056 00000100:00100000:1.0:1789760770.536747:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512712192:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760770.576129:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973953 gen 3 00000100:00100000:1.0:1789760770.576160:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005148700 x1876699512712192/t4294973954(4294973954) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760786 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760770.576196:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760770.576201:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760770.576221:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760770.576248:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760770.576269:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512712192:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760770.577792:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6635, 6635] 00000080:00200000:3.0:1789760770.577808:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760770.577813:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760770.577825:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27181056 i_blocks 53255 00000020:00200000:3.0:1789760770.577829:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760770.577838:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760770.577864:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27181056, count: 4096 00000080:00200000:3.0:1789760770.577871:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27181056, bytes: 4096 00000080:00200000:3.0:1789760770.577880:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760770.577886:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760770.577889:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27181056, 27185152) 0 0 00000080:00200000:3.0:1789760770.577893:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760770.577933:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6636, 6636] 00020000:00200000:3.0:1789760770.577939:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27181056 chunk: [27181056, 27185152) 27185152, 4096 00020000:00200000:3.0:1789760770.577943:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760770.577958:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6636, 6636] 00000008:00010000:3.0:1789760770.577971:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760770.577975:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760770.577985:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27181056, 27185152) append 0 append lockpos 0 00000080:00200000:3.0:1789760770.577995:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27181056=0x19ec000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760770.578037:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760770.578039:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27185152 00000020:00010000:3.0:1789760770.578046:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760770.578049:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.578052:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.578055:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760770.578058:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760770.578063:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760770.578069:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760770.578083:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760770.578160:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 27181056<>27185152 00000100:00100000:3.0:1789760770.578214:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760770.578230:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512712320:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760770.621495:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973955 gen 3 00000008:00010000:3.0:1789760770.621541:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760770.621548:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.621557:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760770.621583:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760770.621619:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512712320:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760770.621760:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6636, 6636] 00000080:00200000:1.0:1789760770.621773:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760770.621778:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760770.621790:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27185152 i_blocks 53255 00000020:00200000:1.0:1789760770.621794:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760770.621803:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760770.621827:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27185152, count: 4096 00000080:00200000:1.0:1789760770.621833:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27185152, bytes: 4096 00000080:00200000:1.0:1789760770.621841:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760770.621849:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760770.621892:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27185152, 27189248) 0 0 00000080:00200000:1.0:1789760770.621897:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760770.621940:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6637, 6637] 00020000:00200000:1.0:1789760770.621948:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27185152 chunk: [27185152, 27189248) 27189248, 4096 00020000:00200000:1.0:1789760770.621951:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760770.621966:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6637, 6637] 00000008:00010000:1.0:1789760770.621981:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760770.621985:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760770.621994:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27185152, 27189248) append 0 append lockpos 0 00000080:00200000:1.0:1789760770.622004:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27185152=0x19ed000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760770.622049:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760770.622052:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27189248 00000020:00010000:1.0:1789760770.622059:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760770.622062:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.622064:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.622068:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760770.622070:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760770.622076:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760770.622082:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760770.622096:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760770.622170:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95c00 - object 0x240000400:5 offset 27185152<>27189248 00000100:00100000:0.0:1789760770.622470:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512712448:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760770.673555:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760770.673567:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760770.673577:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760770.673606:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760770.673630:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512712448:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760770.673730:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6637, 6637] 00000080:00200000:1.0:1789760770.673744:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760770.673750:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760770.673762:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27189248 i_blocks 53255 00000020:00200000:1.0:1789760770.673766:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760770.673774:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760770.673810:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27189248, count: 4096 00000080:00200000:1.0:1789760770.673817:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27189248, bytes: 4096 00000080:00200000:1.0:1789760770.673825:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760770.673833:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760770.673835:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27189248, 27193344) 0 0 00000080:00200000:1.0:1789760770.673839:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760770.673854:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6638, 6638] 00020000:00200000:1.0:1789760770.673861:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27189248 chunk: [27189248, 27193344) 27193344, 4096 00020000:00200000:1.0:1789760770.673864:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760770.674012:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6638, 6638] 00000008:00010000:1.0:1789760770.674028:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760770.674033:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760770.674042:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27189248, 27193344) append 0 append lockpos 0 00000080:00200000:1.0:1789760770.674052:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27189248=0x19ee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760770.674095:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760770.674098:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27193344 00000020:00010000:1.0:1789760770.674104:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760770.674107:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.674110:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.674113:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760770.674115:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760770.674121:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760770.674126:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760770.674140:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760770.674225:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 27189248<>27193344 00000100:00100000:0.0:1789760770.674322:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512712576:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760770.702931:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973957 gen 3 00000008:00010000:0.0:1789760770.702975:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760770.702981:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760770.702988:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760770.703006:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760770.703028:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512712576:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760770.703249:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6638, 6638] 00000080:00200000:1.0:1789760770.703262:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760770.703267:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760770.703280:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27193344 i_blocks 53255 00000020:00200000:1.0:1789760770.703283:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760770.703290:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760770.703315:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27193344, count: 4096 00000080:00200000:1.0:1789760770.703322:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27193344, bytes: 4096 00000080:00200000:1.0:1789760770.703329:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760770.703336:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760770.703338:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27193344, 27197440) 0 0 00000080:00200000:1.0:1789760770.703342:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760770.703359:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6639, 6639] 00020000:00200000:1.0:1789760770.703366:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27193344 chunk: [27193344, 27197440) 27197440, 4096 00020000:00200000:1.0:1789760770.703369:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760770.703385:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6639, 6639] 00000008:00010000:1.0:1789760770.703398:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760770.703403:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760770.703413:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27193344, 27197440) append 0 append lockpos 0 00000080:00200000:1.0:1789760770.703423:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27193344=0x19ef000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760770.703464:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760770.703466:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27197440 00000020:00010000:1.0:1789760770.703473:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760770.703476:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.703478:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.703481:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760770.703484:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760770.703489:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760770.703494:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760770.703506:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760770.703572:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 27193344<>27197440 00000100:00100000:0.0:1789760770.703630:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512712704:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760770.734092:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973958 gen 3 00000008:00010000:0.0:1789760770.734129:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760770.734136:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760770.734145:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760770.734167:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760770.734201:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512712704:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760770.734513:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6639, 6639] 00000080:00200000:1.0:1789760770.734528:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760770.734534:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760770.734546:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27197440 i_blocks 53255 00000020:00200000:1.0:1789760770.734549:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760770.734557:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760770.734586:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27197440, count: 4096 00000080:00200000:1.0:1789760770.734592:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27197440, bytes: 4096 00000080:00200000:1.0:1789760770.734609:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760770.734616:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760770.734618:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27197440, 27201536) 0 0 00000080:00200000:1.0:1789760770.734623:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760770.734637:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6640, 6640] 00020000:00200000:1.0:1789760770.734643:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27197440 chunk: [27197440, 27201536) 27201536, 4096 00020000:00200000:1.0:1789760770.734647:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760770.734663:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6640, 6640] 00000008:00010000:1.0:1789760770.734678:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760770.734683:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760770.734693:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27197440, 27201536) append 0 append lockpos 0 00000080:00200000:1.0:1789760770.734703:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27197440=0x19f0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760770.734749:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760770.734752:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27201536 00000020:00010000:1.0:1789760770.734759:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760770.734762:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.734765:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.734768:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760770.734771:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760770.734778:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760770.734784:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760770.734807:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760770.734896:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94700 - object 0x240000400:5 offset 27197440<>27201536 00000100:00100000:0.0:1789760770.735162:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512712832:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760770.780314:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760770.780325:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760770.780333:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760770.780353:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760770.780378:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512712832:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760770.780466:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6640, 6640] 00000080:00200000:1.0:1789760770.780478:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760770.780483:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760770.780494:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27201536 i_blocks 53255 00000020:00200000:1.0:1789760770.780497:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760770.780504:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760770.780530:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27201536, count: 4096 00000080:00200000:1.0:1789760770.780536:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27201536, bytes: 4096 00000080:00200000:1.0:1789760770.780544:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760770.780551:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760770.780553:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27201536, 27205632) 0 0 00000080:00200000:1.0:1789760770.780557:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760770.780569:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6641, 6641] 00020000:00200000:1.0:1789760770.780575:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27201536 chunk: [27201536, 27205632) 27205632, 4096 00020000:00200000:1.0:1789760770.780579:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760770.780594:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6641, 6641] 00000008:00010000:1.0:1789760770.780677:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760770.780681:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760770.780689:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27201536, 27205632) append 0 append lockpos 0 00000080:00200000:1.0:1789760770.780701:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27201536=0x19f1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760770.780744:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760770.780747:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27205632 00000020:00010000:1.0:1789760770.780753:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760770.780756:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.780759:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.780762:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760770.780764:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760770.780770:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760770.780776:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760770.780802:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760770.780897:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 27201536<>27205632 00000100:00100000:0.0:1789760770.781106:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760770.781121:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512712960:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760770.819660:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973959 gen 3 00000100:00100000:0.0:1789760770.819690:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005c95500 x1876699512712960/t4294973960(4294973960) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760786 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760770.819730:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760770.819736:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760770.819743:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760770.819764:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760770.819784:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512712960:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760770.819937:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6641, 6641] 00000080:00200000:1.0:1789760770.819949:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760770.819954:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760770.819965:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27205632 i_blocks 53255 00000020:00200000:1.0:1789760770.819968:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760770.819975:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760770.820001:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27205632, count: 4096 00000080:00200000:1.0:1789760770.820008:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27205632, bytes: 4096 00000080:00200000:1.0:1789760770.820015:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760770.820022:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760770.820024:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27205632, 27209728) 0 0 00000080:00200000:1.0:1789760770.820029:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760770.820043:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6642, 6642] 00020000:00200000:1.0:1789760770.820051:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27205632 chunk: [27205632, 27209728) 27209728, 4096 00020000:00200000:1.0:1789760770.820054:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760770.820069:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6642, 6642] 00000008:00010000:1.0:1789760770.820083:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760770.820088:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760770.820097:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27205632, 27209728) append 0 append lockpos 0 00000080:00200000:1.0:1789760770.820109:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27205632=0x19f2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760770.820155:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760770.820157:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27209728 00000020:00010000:1.0:1789760770.820164:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760770.820167:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.820392:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.820397:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760770.820399:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760770.820407:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760770.820413:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760770.820428:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760770.820500:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94380 - object 0x240000400:5 offset 27205632<>27209728 00000100:00100000:0.0:1789760770.820558:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512713088:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760770.849287:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760770.852077:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973961 gen 3 00000008:00010000:0.0:1789760770.852124:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760770.852130:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760770.852138:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760770.852160:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760770.852181:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512713088:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760770.852246:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6642, 6642] 00000080:00200000:1.0:1789760770.852260:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760770.852265:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760770.852277:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27209728 i_blocks 53255 00000020:00200000:1.0:1789760770.852280:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760770.852288:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760770.852321:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27209728, count: 4096 00000080:00200000:1.0:1789760770.852327:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27209728, bytes: 4096 00000080:00200000:1.0:1789760770.852334:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760770.852340:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760770.852342:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27209728, 27213824) 0 0 00000080:00200000:1.0:1789760770.852346:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760770.852359:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6643, 6643] 00020000:00200000:1.0:1789760770.852365:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27209728 chunk: [27209728, 27213824) 27213824, 4096 00020000:00200000:1.0:1789760770.852369:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760770.852393:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6643, 6643] 00000008:00010000:1.0:1789760770.852407:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760770.852412:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760770.852420:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27209728, 27213824) append 0 append lockpos 0 00000080:00200000:1.0:1789760770.852430:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27209728=0x19f3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760770.852473:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760770.852476:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27213824 00000020:00010000:1.0:1789760770.852493:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760770.852497:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.852500:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.852503:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760770.852505:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760770.852518:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760770.852530:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760770.852544:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760770.852617:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 27209728<>27213824 00000100:00100000:1.0:1789760770.852686:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512713216:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760770.897729:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973962 gen 3 00000008:00010000:1.0:1789760770.897764:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760770.897770:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760770.897778:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760770.897858:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760770.897882:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512713216:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760770.902613:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6643, 6643] 00000080:00200000:3.0:1789760770.902631:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760770.902637:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760770.902650:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27213824 i_blocks 53255 00000020:00200000:3.0:1789760770.902655:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760770.902669:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760770.902695:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27213824, count: 4096 00000080:00200000:3.0:1789760770.902703:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27213824, bytes: 4096 00000080:00200000:3.0:1789760770.902715:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760770.902723:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760770.902725:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27213824, 27217920) 0 0 00000080:00200000:3.0:1789760770.902730:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760770.902744:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6644, 6644] 00020000:00200000:3.0:1789760770.902751:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27213824 chunk: [27213824, 27217920) 27217920, 4096 00020000:00200000:3.0:1789760770.902755:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760770.902772:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6644, 6644] 00000008:00010000:3.0:1789760770.902787:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760770.902792:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760770.902801:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27213824, 27217920) append 0 append lockpos 0 00000080:00200000:3.0:1789760770.902812:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27213824=0x19f4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760770.902865:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760770.902868:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27217920 00000020:00010000:3.0:1789760770.902875:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760770.902878:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.902881:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760770.902885:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760770.902888:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760770.902894:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760770.904151:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760770.904199:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760770.904323:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 27213824<>27217920 00000100:00100000:2.0:1789760770.904427:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512713344:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760770.939462:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760770.939473:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760770.939482:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760770.939505:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760770.939529:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512713344:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760770.940499:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6644, 6644] 00000080:00200000:1.0:1789760770.940513:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760770.940519:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760770.940531:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27217920 i_blocks 53255 00000020:00200000:1.0:1789760770.940535:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760770.940544:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760770.940568:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27217920, count: 4096 00000080:00200000:1.0:1789760770.940575:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27217920, bytes: 4096 00000080:00200000:1.0:1789760770.940583:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760770.940590:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760770.940592:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27217920, 27222016) 0 0 00000080:00200000:1.0:1789760770.940595:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760770.940608:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6645, 6645] 00020000:00200000:1.0:1789760770.940614:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27217920 chunk: [27217920, 27222016) 27222016, 4096 00020000:00200000:1.0:1789760770.940618:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760770.940633:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6645, 6645] 00000008:00010000:1.0:1789760770.940646:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760770.940651:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760770.940659:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27217920, 27222016) append 0 append lockpos 0 00000080:00200000:1.0:1789760770.940669:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27217920=0x19f5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760770.940715:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760770.940717:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27222016 00000020:00010000:1.0:1789760770.940724:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760770.940727:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.940730:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.940733:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760770.940736:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760770.940743:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760770.940748:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760770.940764:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760770.940842:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 27217920<>27222016 00000100:00100000:0.0:1789760770.940941:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512713472:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760770.974264:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973964 gen 3 00000008:00010000:0.0:1789760770.974310:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760770.974317:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760770.974325:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760770.974344:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760770.974368:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512713472:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760770.974459:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6645, 6645] 00000080:00200000:1.0:1789760770.974471:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760770.974476:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760770.974488:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27222016 i_blocks 53255 00000020:00200000:1.0:1789760770.974491:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760770.974500:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760770.974528:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27222016, count: 4096 00000080:00200000:1.0:1789760770.974535:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27222016, bytes: 4096 00000080:00200000:1.0:1789760770.974542:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760770.974549:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760770.974552:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27222016, 27226112) 0 0 00000080:00200000:1.0:1789760770.974555:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760770.974568:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6646, 6646] 00020000:00200000:1.0:1789760770.974574:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27222016 chunk: [27222016, 27226112) 27226112, 4096 00020000:00200000:1.0:1789760770.974578:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760770.974591:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6646, 6646] 00000008:00010000:1.0:1789760770.974616:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760770.974620:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760770.974628:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27222016, 27226112) append 0 append lockpos 0 00000080:00200000:1.0:1789760770.974639:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27222016=0x19f6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760770.974683:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760770.974686:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27226112 00000020:00010000:1.0:1789760770.974692:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760770.974696:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.974698:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760770.974701:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760770.974704:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760770.974710:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760770.974716:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760770.974730:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760770.974805:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 27222016<>27226112 00000100:00100000:0.0:1789760770.974950:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512713600:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760771.012486:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973965 gen 3 00000008:00010000:0.0:1789760771.012522:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760771.012529:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760771.012537:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760771.012557:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760771.012580:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512713600:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760771.012981:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6646, 6646] 00000080:00200000:1.0:1789760771.012992:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760771.012997:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760771.013009:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27226112 i_blocks 53255 00000020:00200000:1.0:1789760771.013013:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760771.013022:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760771.013048:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27226112, count: 4096 00000080:00200000:1.0:1789760771.013054:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27226112, bytes: 4096 00000080:00200000:1.0:1789760771.013061:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760771.013068:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760771.013070:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27226112, 27230208) 0 0 00000080:00200000:1.0:1789760771.013074:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760771.013087:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6647, 6647] 00020000:00200000:1.0:1789760771.013094:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27226112 chunk: [27226112, 27230208) 27230208, 4096 00020000:00200000:1.0:1789760771.013098:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760771.013111:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6647, 6647] 00000008:00010000:1.0:1789760771.013124:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760771.013129:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760771.013137:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27226112, 27230208) append 0 append lockpos 0 00000080:00200000:1.0:1789760771.013149:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27226112=0x19f7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760771.013193:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760771.013196:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27230208 00000020:00010000:1.0:1789760771.013210:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760771.013214:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760771.013217:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760771.013220:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760771.013222:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760771.013229:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760771.013234:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760771.013249:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760771.013329:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 27226112<>27230208 00000100:00100000:0.0:1789760771.013418:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760771.013432:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512713728:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760771.043111:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973966 gen 3 00000008:00010000:0.0:1789760771.043147:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760771.043154:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760771.043162:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760771.043182:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760771.043217:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512713728:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760771.043385:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6647, 6647] 00000080:00200000:1.0:1789760771.043398:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760771.043403:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760771.043413:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27230208 i_blocks 53255 00000020:00200000:1.0:1789760771.043417:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760771.043425:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760771.043454:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27230208, count: 4096 00000080:00200000:1.0:1789760771.043461:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27230208, bytes: 4096 00000080:00200000:1.0:1789760771.043469:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760771.043475:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760771.043477:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27230208, 27234304) 0 0 00000080:00200000:1.0:1789760771.043481:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760771.043494:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6648, 6648] 00020000:00200000:1.0:1789760771.043501:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27230208 chunk: [27230208, 27234304) 27234304, 4096 00020000:00200000:1.0:1789760771.043505:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760771.043521:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6648, 6648] 00000008:00010000:1.0:1789760771.043534:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760771.043539:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760771.043548:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27230208, 27234304) append 0 append lockpos 0 00000080:00200000:1.0:1789760771.043558:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27230208=0x19f8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760771.043614:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760771.043616:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27234304 00000020:00010000:1.0:1789760771.043623:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760771.043626:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760771.043629:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760771.043631:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760771.043634:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760771.043639:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760771.043645:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760771.043659:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760771.043736:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97480 - object 0x240000400:5 offset 27230208<>27234304 00000100:00100000:0.0:1789760771.043798:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512713856:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760771.078605:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760771.078616:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760771.078625:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760771.078648:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760771.078671:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512713856:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760771.078760:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6648, 6648] 00000080:00200000:1.0:1789760771.078771:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760771.078777:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760771.078798:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27234304 i_blocks 53255 00000020:00200000:1.0:1789760771.078802:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760771.078811:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760771.078834:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27234304, count: 4096 00000080:00200000:1.0:1789760771.078840:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27234304, bytes: 4096 00000080:00200000:1.0:1789760771.078848:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760771.078855:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760771.078857:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27234304, 27238400) 0 0 00000080:00200000:1.0:1789760771.078861:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760771.078873:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6649, 6649] 00020000:00200000:1.0:1789760771.078888:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27234304 chunk: [27234304, 27238400) 27238400, 4096 00020000:00200000:1.0:1789760771.078893:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760771.078934:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6649, 6649] 00000008:00010000:1.0:1789760771.078949:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760771.078954:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760771.078963:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27234304, 27238400) append 0 append lockpos 0 00000080:00200000:1.0:1789760771.078973:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27234304=0x19f9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760771.079014:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760771.079016:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27238400 00000020:00010000:1.0:1789760771.079022:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760771.079025:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760771.079028:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760771.079030:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760771.079033:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760771.079039:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760771.079044:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760771.079057:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760771.079127:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96680 - object 0x240000400:5 offset 27234304<>27238400 00000100:00100000:0.0:1789760771.079261:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512713984:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760771.123006:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973968 gen 3 00000008:00010000:0.0:1789760771.123061:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760771.123069:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760771.123078:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760771.123103:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760771.123128:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512713984:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760771.124192:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6649, 6649] 00000080:00200000:1.0:1789760771.124220:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760771.124226:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760771.124240:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27238400 i_blocks 53255 00000020:00200000:1.0:1789760771.124244:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760771.124255:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760771.124283:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27238400, count: 4096 00000080:00200000:1.0:1789760771.124290:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27238400, bytes: 4096 00000080:00200000:1.0:1789760771.124301:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760771.124309:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760771.124311:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27238400, 27242496) 0 0 00000080:00200000:1.0:1789760771.124315:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760771.124330:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6650, 6650] 00020000:00200000:1.0:1789760771.124337:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27238400 chunk: [27238400, 27242496) 27242496, 4096 00020000:00200000:1.0:1789760771.124341:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760771.124363:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6650, 6650] 00000008:00010000:1.0:1789760771.124380:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760771.124385:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760771.124395:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27238400, 27242496) append 0 append lockpos 0 00000080:00200000:1.0:1789760771.124406:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27238400=0x19fa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760771.124453:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760771.124456:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27242496 00000020:00010000:1.0:1789760771.124462:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760771.124465:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760771.124468:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760771.124471:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760771.124474:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760771.124480:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760771.124486:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760771.124503:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760771.124589:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95880 - object 0x240000400:5 offset 27238400<>27242496 00000100:00100000:0.0:1789760771.124783:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512714112:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760771.189514:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973969 gen 3 00000008:00010000:0.0:1789760771.189558:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760771.189564:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760771.189573:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760771.189643:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760771.189666:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512714112:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760771.190087:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6650, 6650] 00000080:00200000:1.0:1789760771.190101:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760771.190106:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760771.190118:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27242496 i_blocks 53255 00000020:00200000:1.0:1789760771.190121:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760771.190231:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760771.190259:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27242496, count: 4096 00000080:00200000:1.0:1789760771.190265:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27242496, bytes: 4096 00000080:00200000:1.0:1789760771.190273:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760771.190280:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760771.190282:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27242496, 27246592) 0 0 00000080:00200000:1.0:1789760771.190286:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760771.190298:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6651, 6651] 00020000:00200000:1.0:1789760771.190304:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27242496 chunk: [27242496, 27246592) 27246592, 4096 00020000:00200000:1.0:1789760771.190307:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760771.190323:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6651, 6651] 00000008:00010000:1.0:1789760771.190337:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760771.190342:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760771.190352:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27242496, 27246592) append 0 append lockpos 0 00000080:00200000:1.0:1789760771.190363:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27242496=0x19fb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760771.190411:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760771.190414:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27246592 00000020:00010000:1.0:1789760771.190421:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760771.190424:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760771.190427:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760771.190430:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760771.190432:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760771.190439:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760771.190444:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760771.190461:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760771.190540:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96d80 - object 0x240000400:5 offset 27242496<>27246592 00000100:00100000:0.0:1789760771.191238:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512714240:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760771.243625:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973970 gen 3 00000008:00010000:0.0:1789760771.243661:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760771.243667:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760771.243676:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760771.243702:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760771.243726:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512714240:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760771.244253:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6651, 6651] 00000080:00200000:3.0:1789760771.244274:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760771.244280:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760771.244291:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27246592 i_blocks 53255 00000020:00200000:3.0:1789760771.244295:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760771.244303:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760771.244330:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27246592, count: 4096 00000080:00200000:3.0:1789760771.244336:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27246592, bytes: 4096 00000080:00200000:3.0:1789760771.244345:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760771.244352:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760771.244354:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27246592, 27250688) 0 0 00000080:00200000:3.0:1789760771.244358:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760771.244371:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6652, 6652] 00020000:00200000:3.0:1789760771.244378:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27246592 chunk: [27246592, 27250688) 27250688, 4096 00020000:00200000:3.0:1789760771.244382:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760771.244398:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6652, 6652] 00000008:00010000:3.0:1789760771.244412:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760771.244417:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760771.244425:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27246592, 27250688) append 0 append lockpos 0 00000080:00200000:3.0:1789760771.244436:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27246592=0x19fc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760771.244491:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760771.244494:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27250688 00000020:00010000:3.0:1789760771.244501:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760771.244504:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760771.244507:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760771.244510:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760771.244513:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760771.244519:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760771.244524:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760771.244537:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760771.244778:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107afb80 - object 0x240000400:5 offset 27246592<>27250688 00000100:00100000:3.0:1789760771.244824:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760771.244840:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512714368:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760771.319169:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973971 gen 3 00000008:00010000:3.0:1789760771.319230:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760771.319238:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.319251:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.319286:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760771.319316:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512714368:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760771.319515:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6652, 6652] 00000080:00200000:0.0:1789760771.319532:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.319538:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.319554:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27250688 i_blocks 53255 00000020:00200000:0.0:1789760771.319558:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.319569:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.319614:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27250688, count: 4096 00000080:00200000:0.0:1789760771.319623:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27250688, bytes: 4096 00000080:00200000:0.0:1789760771.319636:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.319646:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.319648:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27250688, 27254784) 0 0 00000080:00200000:0.0:1789760771.319653:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.319841:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6653, 6653] 00020000:00200000:0.0:1789760771.319851:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27250688 chunk: [27250688, 27254784) 27254784, 4096 00020000:00200000:0.0:1789760771.319856:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.319889:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6653, 6653] 00000008:00010000:0.0:1789760771.319946:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.319953:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760771.319968:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27250688, 27254784) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.319983:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27250688=0x19fd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.320048:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.320051:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27254784 00000020:00010000:0.0:1789760771.320059:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.320063:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.320066:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.320069:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.320073:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.320080:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.320087:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.320107:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccc00 00000008:00100000:0.0:1789760771.320233:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514aa00 - object 0x240000400:5 offset 27250688<>27254784 00000100:00100000:1.0:1789760771.320329:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512714496:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760771.398575:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973972 gen 3 00000008:00010000:1.0:1789760771.398647:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760771.398664:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.398679:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.398694:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760771.398733:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512714496:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760771.399360:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6653, 6653] 00000080:00200000:0.0:1789760771.399378:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.399386:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.399403:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27254784 i_blocks 53255 00000020:00200000:0.0:1789760771.399407:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.399419:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.399455:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27254784, count: 4096 00000080:00200000:0.0:1789760771.399463:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27254784, bytes: 4096 00000080:00200000:0.0:1789760771.399487:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.399495:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.399498:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27254784, 27258880) 0 0 00000080:00200000:0.0:1789760771.399503:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.399522:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6654, 6654] 00020000:00200000:0.0:1789760771.399531:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27254784 chunk: [27254784, 27258880) 27258880, 4096 00020000:00200000:0.0:1789760771.399535:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.399557:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6654, 6654] 00000008:00010000:0.0:1789760771.399574:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.399581:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760771.399750:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27254784, 27258880) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.399770:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27254784=0x19fe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.399838:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.399841:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27258880 00000020:00010000:0.0:1789760771.399850:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.399854:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.399857:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.399860:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.399864:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.399872:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.399880:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.400302:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cce40 00000008:00100000:0.0:1789760771.400424:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514aa00 - object 0x240000400:5 offset 27254784<>27258880 00000100:00100000:1.0:1789760771.401011:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512714624:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760771.499731:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973973 gen 3 00000008:00010000:1.0:1789760771.499784:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760771.499802:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.499814:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.499842:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760771.499890:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512714624:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760771.500353:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6654, 6654] 00000080:00200000:0.0:1789760771.500373:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.500382:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.500404:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27258880 i_blocks 53255 00000020:00200000:0.0:1789760771.500410:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.500422:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.500471:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27258880, count: 4096 00000080:00200000:0.0:1789760771.500480:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27258880, bytes: 4096 00000080:00200000:0.0:1789760771.500495:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.500506:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.500508:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27258880, 27262976) 0 0 00000080:00200000:0.0:1789760771.500514:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.500538:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6655, 6655] 00020000:00200000:0.0:1789760771.500671:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27258880 chunk: [27258880, 27262976) 27262976, 4096 00020000:00200000:0.0:1789760771.500676:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.500700:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6655, 6655] 00000008:00010000:0.0:1789760771.500719:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.500726:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760771.500737:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27258880, 27262976) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.500752:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27258880=0x19ff000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.500830:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.500833:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27262976 00000020:00010000:0.0:1789760771.500842:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.500846:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.500849:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.500853:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.500856:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.500864:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.501057:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.501080:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cce40 00000008:00100000:0.0:1789760771.501195:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148700 - object 0x240000400:5 offset 27258880<>27262976 00000100:00100000:1.0:1789760771.501429:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512714752:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760771.562406:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760771.562423:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.562436:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.562461:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760771.562496:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512714752:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760771.562736:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6655, 6655] 00000080:00200000:0.0:1789760771.562753:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.562760:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.562776:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27262976 i_blocks 53255 00000020:00200000:0.0:1789760771.562779:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.562790:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.562879:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27262976, count: 4096 00000080:00200000:0.0:1789760771.562888:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27262976, bytes: 4096 00000080:00200000:0.0:1789760771.563327:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.563338:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.563340:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27262976, 27267072) 0 0 00000080:00200000:0.0:1789760771.563346:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.563364:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6656, 6656] 00020000:00200000:0.0:1789760771.563440:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27262976 chunk: [27262976, 27267072) 27267072, 4096 00020000:00200000:0.0:1789760771.563444:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.563467:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6656, 6656] 00000008:00010000:0.0:1789760771.563486:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.563491:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760771.563502:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27262976, 27267072) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.563516:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27262976=0x1a00000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.563577:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.563580:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27267072 00000020:00010000:0.0:1789760771.564103:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.564110:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.564114:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.564118:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.564121:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.564132:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.564142:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.564169:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccc00 00000008:00100000:0.0:1789760771.564303:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514ad80 - object 0x240000400:5 offset 27262976<>27267072 00000100:00100000:1.0:1789760771.564489:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512714880:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760771.611216:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973975 gen 3 00000008:00010000:1.0:1789760771.611292:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760771.611301:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.611313:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.611335:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760771.611365:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512714880:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760771.611471:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6656, 6656] 00000080:00200000:0.0:1789760771.611487:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.611494:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.611511:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27267072 i_blocks 53255 00000020:00200000:0.0:1789760771.611515:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.611525:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.611559:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27267072, count: 4096 00000080:00200000:0.0:1789760771.611566:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27267072, bytes: 4096 00000080:00200000:0.0:1789760771.611574:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.611582:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.611584:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27267072, 27271168) 0 0 00000080:00200000:0.0:1789760771.611587:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.611605:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6657, 6657] 00020000:00200000:0.0:1789760771.611613:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27267072 chunk: [27267072, 27271168) 27271168, 4096 00020000:00200000:0.0:1789760771.611617:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.611638:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6657, 6657] 00000008:00010000:0.0:1789760771.611656:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.611662:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760771.611674:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27267072, 27271168) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.611688:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27267072=0x1a01000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.611738:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.611741:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27271168 00000020:00010000:0.0:1789760771.611748:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.611750:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.611752:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.611755:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.611757:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.611762:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.611767:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.611781:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000008:00100000:0.0:1789760771.611885:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148a80 - object 0x240000400:5 offset 27267072<>27271168 00000100:00100000:1.0:1789760771.612264:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760771.612284:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512715008:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760771.641631:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973976 gen 3 00000008:00010000:1.0:1789760771.641679:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760771.641688:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.641700:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.641726:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760771.641755:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512715008:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760771.642823:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6657, 6657] 00000080:00200000:0.0:1789760771.642845:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.642852:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.642871:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27271168 i_blocks 55303 00000020:00200000:0.0:1789760771.642885:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.642896:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.642986:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27271168, count: 4096 00000080:00200000:0.0:1789760771.643003:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27271168, bytes: 4096 00000080:00200000:0.0:1789760771.643019:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.643028:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.643030:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27271168, 27275264) 0 0 00000080:00200000:0.0:1789760771.643035:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.643054:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6658, 6658] 00020000:00200000:0.0:1789760771.643063:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27271168 chunk: [27271168, 27275264) 27275264, 4096 00020000:00200000:0.0:1789760771.643067:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.643100:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6658, 6658] 00000008:00010000:0.0:1789760771.643119:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.643125:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760771.643137:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27271168, 27275264) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.643152:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27271168=0x1a02000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.643232:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.643235:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27275264 00000020:00010000:0.0:1789760771.643243:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.643247:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.643250:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.643253:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.643256:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.643273:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.643281:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.643304:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc180 00000008:00100000:0.0:1789760771.643421:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148e00 - object 0x240000400:5 offset 27271168<>27275264 00000100:00100000:1.0:1789760771.643699:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760771.643724:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512715136:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760771.671748:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973977 gen 3 00000008:00010000:1.0:1789760771.671853:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760771.671860:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.671950:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.671974:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760771.671996:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512715136:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760771.672101:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6658, 6658] 00000080:00200000:0.0:1789760771.672114:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.672121:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.672134:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27275264 i_blocks 55303 00000020:00200000:0.0:1789760771.672138:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.672145:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.672172:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27275264, count: 4096 00000080:00200000:0.0:1789760771.672178:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27275264, bytes: 4096 00000080:00200000:0.0:1789760771.672185:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.672192:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.672195:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27275264, 27279360) 0 0 00000080:00200000:0.0:1789760771.672209:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.672226:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6659, 6659] 00020000:00200000:0.0:1789760771.672233:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27275264 chunk: [27275264, 27279360) 27279360, 4096 00020000:00200000:0.0:1789760771.672237:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.672251:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6659, 6659] 00000008:00010000:0.0:1789760771.672265:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.672269:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760771.672279:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27275264, 27279360) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.672289:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27275264=0x1a03000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.672331:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.672334:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27279360 00000020:00010000:0.0:1789760771.672341:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.672344:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.672347:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.672350:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.672353:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.672359:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.672364:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.672378:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccd80 00000008:00100000:0.0:1789760771.672455:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a680 - object 0x240000400:5 offset 27275264<>27279360 00000100:00100000:1.0:1789760771.672839:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760771.672855:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512715264:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760771.710034:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973978 gen 3 00000008:00010000:1.0:1789760771.710069:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760771.710075:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.710085:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.710107:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760771.710131:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512715264:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760771.710247:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6659, 6659] 00000080:00200000:0.0:1789760771.710261:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.710266:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.710279:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27279360 i_blocks 55303 00000020:00200000:0.0:1789760771.710283:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.710291:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.710317:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27279360, count: 4096 00000080:00200000:0.0:1789760771.710323:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27279360, bytes: 4096 00000080:00200000:0.0:1789760771.710332:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.710339:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.710341:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27279360, 27283456) 0 0 00000080:00200000:0.0:1789760771.710345:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.710357:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6660, 6660] 00020000:00200000:0.0:1789760771.710363:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27279360 chunk: [27279360, 27283456) 27283456, 4096 00020000:00200000:0.0:1789760771.710367:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.710381:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6660, 6660] 00000008:00010000:0.0:1789760771.710393:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.710398:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760771.710407:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27279360, 27283456) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.710417:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27279360=0x1a04000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.710459:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.710462:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27283456 00000020:00010000:0.0:1789760771.710468:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.710472:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.710475:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.710478:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.710481:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.710488:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.710493:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.710506:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc780 00000008:00100000:0.0:1789760771.712179:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a300 - object 0x240000400:5 offset 27279360<>27283456 00000100:00100000:1.0:1789760771.712389:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760771.712410:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512715392:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760771.765259:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973979 gen 3 00000008:00010000:1.0:1789760771.765297:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760771.765305:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.765314:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.765333:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760771.765354:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512715392:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760771.765881:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6660, 6660] 00000080:00200000:0.0:1789760771.765894:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.765924:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.765936:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27283456 i_blocks 55303 00000020:00200000:0.0:1789760771.765939:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.765947:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.765969:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27283456, count: 4096 00000080:00200000:0.0:1789760771.765975:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27283456, bytes: 4096 00000080:00200000:0.0:1789760771.765983:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.765990:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.765992:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27283456, 27287552) 0 0 00000080:00200000:0.0:1789760771.765995:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.766007:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6661, 6661] 00020000:00200000:0.0:1789760771.766013:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27283456 chunk: [27283456, 27287552) 27287552, 4096 00020000:00200000:0.0:1789760771.766017:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.766030:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6661, 6661] 00000008:00010000:0.0:1789760771.766043:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.766047:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760771.766055:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27283456, 27287552) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.766065:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27283456=0x1a05000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.766102:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.766105:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27287552 00000020:00010000:0.0:1789760771.766111:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.766114:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.766116:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.766119:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.766122:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.766128:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.766133:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.766147:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000008:00100000:0.0:1789760771.766236:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149880 - object 0x240000400:5 offset 27283456<>27287552 00000100:00100000:1.0:1789760771.766293:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512715520:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760771.807140:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973980 gen 3 00000008:00010000:1.0:1789760771.807174:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760771.807180:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.807189:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.807217:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760771.807240:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512715520:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760771.807335:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6661, 6661] 00000080:00200000:0.0:1789760771.807347:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.807352:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.807363:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27287552 i_blocks 55303 00000020:00200000:0.0:1789760771.807367:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.807374:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.807400:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27287552, count: 4096 00000080:00200000:0.0:1789760771.807406:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27287552, bytes: 4096 00000080:00200000:0.0:1789760771.807424:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.807431:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.807434:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27287552, 27291648) 0 0 00000080:00200000:0.0:1789760771.807437:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.807451:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6662, 6662] 00020000:00200000:0.0:1789760771.807458:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27287552 chunk: [27287552, 27291648) 27291648, 4096 00020000:00200000:0.0:1789760771.807462:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.807476:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6662, 6662] 00000008:00010000:0.0:1789760771.807497:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.807502:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760771.807511:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27287552, 27291648) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.807521:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27287552=0x1a06000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.807562:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.807564:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27291648 00000020:00010000:0.0:1789760771.807570:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.807573:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.807576:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.807578:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.807581:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.807587:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.807591:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.807603:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc000 00000008:00100000:0.0:1789760771.807685:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b480 - object 0x240000400:5 offset 27287552<>27291648 00000100:00100000:1.0:1789760771.807870:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512715648:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760771.855118:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760771.855130:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760771.855140:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760771.855150:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760771.855172:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512715648:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760771.855334:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6662, 6662] 00000080:00200000:0.0:1789760771.855347:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.855353:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.855374:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27291648 i_blocks 55303 00000020:00200000:0.0:1789760771.855378:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.855386:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.855412:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27291648, count: 4096 00000080:00200000:0.0:1789760771.855418:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27291648, bytes: 4096 00000080:00200000:0.0:1789760771.855426:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.855432:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.855434:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27291648, 27295744) 0 0 00000080:00200000:0.0:1789760771.855438:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.855451:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6663, 6663] 00020000:00200000:0.0:1789760771.855458:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27291648 chunk: [27291648, 27295744) 27295744, 4096 00020000:00200000:0.0:1789760771.855461:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.855475:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6663, 6663] 00000008:00010000:0.0:1789760771.855489:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.855494:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760771.855503:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27291648, 27295744) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.855514:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27291648=0x1a07000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.855894:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.855898:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27295744 00000020:00010000:0.0:1789760771.855936:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.855939:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.855942:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.855945:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.855948:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.855954:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.855960:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.855974:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccf00 00000008:00100000:0.0:1789760771.856053:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148380 - object 0x240000400:5 offset 27291648<>27295744 00000100:00100000:0.0:1789760771.856084:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760771.856097:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512715776:192.168.203.156@tcp:4:dd.0 00010000:00010000:0.0:1789760771.869275:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760771.895278:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973982 gen 3 00000008:00010000:0.0:1789760771.895333:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:0.0:1789760771.895340:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760771.895349:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760771.895372:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:0.0:1789760771.895394:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512715776:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760771.895533:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6663, 6663] 00000080:00200000:3.0:1789760771.895544:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760771.895548:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760771.895557:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27295744 i_blocks 55303 00000020:00200000:3.0:1789760771.895560:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760771.895566:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760771.895602:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27295744, count: 4096 00000080:00200000:3.0:1789760771.895609:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27295744, bytes: 4096 00000080:00200000:3.0:1789760771.895617:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760771.895624:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760771.895625:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27295744, 27299840) 0 0 00000080:00200000:3.0:1789760771.895629:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760771.895643:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6664, 6664] 00020000:00200000:3.0:1789760771.895649:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27295744 chunk: [27295744, 27299840) 27299840, 4096 00020000:00200000:3.0:1789760771.895653:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760771.895667:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6664, 6664] 00000008:00010000:3.0:1789760771.895682:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760771.895686:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760771.895695:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27295744, 27299840) append 0 append lockpos 0 00000080:00200000:3.0:1789760771.895706:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27295744=0x1a08000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760771.895750:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760771.895765:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27299840 00000020:00010000:3.0:1789760771.895772:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760771.895775:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760771.895778:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760771.895781:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760771.895783:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760771.895789:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760771.895794:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760771.895807:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760771.895881:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107afb80 - object 0x240000400:5 offset 27295744<>27299840 00000100:00100000:2.0:1789760771.897152:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512715904:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760771.933700:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973983 gen 3 00000008:00010000:3.0:1789760771.933745:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760771.933752:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.933762:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.933785:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000080:00200000:0.0:1789760771.933964:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6664, 6664] 00000080:00200000:0.0:1789760771.933979:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.933985:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.933996:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27299840 i_blocks 55303 00000020:00200000:0.0:1789760771.934000:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.934008:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.934033:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27299840, count: 4096 00000080:00200000:0.0:1789760771.934040:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27299840, bytes: 4096 00000080:00200000:0.0:1789760771.934049:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.934056:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.934057:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27299840, 27303936) 0 0 00000080:00200000:0.0:1789760771.934062:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.934075:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6665, 6665] 00020000:00200000:0.0:1789760771.934081:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27299840 chunk: [27299840, 27303936) 27303936, 4096 00020000:00200000:0.0:1789760771.934085:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.934102:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6665, 6665] 00000008:00010000:0.0:1789760771.934115:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.934120:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760771.934130:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27299840, 27303936) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.934140:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27299840=0x1a09000, pages 1 (max 131072), lockless, parallel 00000100:00100000:3.0:1789760771.934241:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512715904:192.168.203.156@tcp:4:dd.0 00000020:00200000:0.0:1789760771.934330:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.934332:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27303936 00000008:00010000:3.0:1789760771.934333:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00010000:0.0:1789760771.934340:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.934344:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.934347:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.934349:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.934352:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.934358:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.934365:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00100000:3.0:1789760771.934417:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107afb80 - object 0x240000400:5 offset 27299840<>27303936 00000100:00100000:3.0:1789760771.934450:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760771.934544:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512716032:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760771.989519:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973984 gen 3 00000008:00010000:3.0:1789760771.989555:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:3.0:1789760771.989562:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760771.989571:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760771.989593:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:3.0:1789760771.989616:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512716032:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760771.989832:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6665, 6665] 00000080:00200000:0.0:1789760771.989846:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760771.989851:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760771.989862:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27303936 i_blocks 55303 00000020:00200000:0.0:1789760771.989865:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760771.989873:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760771.989927:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27303936, count: 4096 00000080:00200000:0.0:1789760771.989934:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27303936, bytes: 4096 00000080:00200000:0.0:1789760771.989943:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760771.989949:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760771.989951:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27303936, 27308032) 0 0 00000080:00200000:0.0:1789760771.989955:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760771.989968:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6666, 6666] 00020000:00200000:0.0:1789760771.989976:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27303936 chunk: [27303936, 27308032) 27308032, 4096 00020000:00200000:0.0:1789760771.989980:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760771.989995:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6666, 6666] 00000008:00010000:0.0:1789760771.990009:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760771.990014:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760771.990022:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27303936, 27308032) append 0 append lockpos 0 00000080:00200000:0.0:1789760771.990033:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27303936=0x1a0a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760771.990081:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760771.990083:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27308032 00000020:00010000:0.0:1789760771.990090:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760771.990094:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.990097:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760771.990100:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760771.990103:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760771.990109:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760771.990114:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760771.990128:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cca80 00000008:00100000:0.0:1789760771.990212:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148380 - object 0x240000400:5 offset 27303936<>27308032 00000100:00100000:1.0:1789760771.990357:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512716160:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760772.020004:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760772.020016:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.020026:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.020099:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760772.020124:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512716160:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760772.020457:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6666, 6666] 00000080:00200000:0.0:1789760772.020470:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.020476:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.020486:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27308032 i_blocks 55303 00000020:00200000:0.0:1789760772.020490:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.020498:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.020525:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27308032, count: 4096 00000080:00200000:0.0:1789760772.020664:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27308032, bytes: 4096 00000080:00200000:0.0:1789760772.020779:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.020786:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.020788:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27308032, 27312128) 0 0 00000080:00200000:0.0:1789760772.020792:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.020805:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6667, 6667] 00020000:00200000:0.0:1789760772.020811:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27308032 chunk: [27308032, 27312128) 27312128, 4096 00020000:00200000:0.0:1789760772.020815:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.020831:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6667, 6667] 00000008:00010000:0.0:1789760772.020845:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.020951:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760772.021050:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27308032, 27312128) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.021064:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27308032=0x1a0b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.021111:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.021114:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27312128 00000020:00010000:0.0:1789760772.021120:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.021123:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.021126:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.021129:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.021132:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.021139:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.021144:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.021155:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc540 00000008:00100000:0.0:1789760772.021243:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b480 - object 0x240000400:5 offset 27308032<>27312128 00000100:00100000:1.0:1789760772.021359:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512716288:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760772.068117:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973985 gen 3 00000100:00100000:1.0:1789760772.068146:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00514b480 x1876699512716288/t4294973986(4294973986) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760788 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760772.068182:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760772.068188:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.068204:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.068224:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760772.068243:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512716288:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760772.068468:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6667, 6667] 00000080:00200000:0.0:1789760772.068480:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.068484:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.068496:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27312128 i_blocks 55303 00000020:00200000:0.0:1789760772.068499:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.068507:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.068534:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27312128, count: 4096 00000080:00200000:0.0:1789760772.068539:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27312128, bytes: 4096 00000080:00200000:0.0:1789760772.068546:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.068554:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.068556:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27312128, 27316224) 0 0 00000080:00200000:0.0:1789760772.068559:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.068572:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6668, 6668] 00020000:00200000:0.0:1789760772.068578:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27312128 chunk: [27312128, 27316224) 27316224, 4096 00020000:00200000:0.0:1789760772.068581:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.068662:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6668, 6668] 00000008:00010000:0.0:1789760772.068677:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.068682:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760772.068692:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27312128, 27316224) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.068703:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27312128=0x1a0c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.068744:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.068746:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27316224 00000020:00010000:0.0:1789760772.068753:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.068756:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.068759:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.068763:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.068766:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.068772:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.068777:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.068790:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc600 00000008:00100000:0.0:1789760772.068866:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b800 - object 0x240000400:5 offset 27312128<>27316224 00000100:00100000:0.0:1789760772.069179:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760772.069192:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512716416:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760772.126818:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973987 gen 3 00000008:00010000:0.0:1789760772.126865:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:0.0:1789760772.126872:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760772.126880:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760772.126973:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:0.0:1789760772.126996:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512716416:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760772.127065:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6668, 6668] 00000080:00200000:1.0:1789760772.127077:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760772.127082:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760772.127094:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27316224 i_blocks 55303 00000020:00200000:1.0:1789760772.127289:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760772.127299:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760772.127324:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27316224, count: 4096 00000080:00200000:1.0:1789760772.127330:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27316224, bytes: 4096 00000080:00200000:1.0:1789760772.127340:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760772.127346:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760772.127348:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27316224, 27320320) 0 0 00000080:00200000:1.0:1789760772.127352:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760772.127363:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6669, 6669] 00020000:00200000:1.0:1789760772.127369:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27316224 chunk: [27316224, 27320320) 27320320, 4096 00020000:00200000:1.0:1789760772.127374:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760772.127389:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6669, 6669] 00000008:00010000:1.0:1789760772.127403:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760772.127407:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760772.127418:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27316224, 27320320) append 0 append lockpos 0 00000080:00200000:1.0:1789760772.127429:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27316224=0x1a0d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760772.127474:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760772.127477:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27320320 00000020:00010000:1.0:1789760772.127484:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760772.127486:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.127489:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.127492:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760772.127494:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760772.127500:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760772.127506:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760772.127519:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760772.127599:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96d80 - object 0x240000400:5 offset 27316224<>27320320 00000100:00100000:1.0:1789760772.127630:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760772.127641:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512716544:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760772.175822:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973988 gen 3 00000008:00010000:1.0:1789760772.175858:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760772.175864:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.175872:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.175893:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760772.175946:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512716544:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760772.176033:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6669, 6669] 00000080:00200000:3.0:1789760772.176045:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760772.176050:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760772.176061:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27320320 i_blocks 55303 00000020:00200000:3.0:1789760772.176064:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760772.176072:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760772.176098:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27320320, count: 4096 00000080:00200000:3.0:1789760772.176104:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27320320, bytes: 4096 00000080:00200000:3.0:1789760772.176111:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760772.176118:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760772.176119:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27320320, 27324416) 0 0 00000080:00200000:3.0:1789760772.176124:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760772.176136:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6670, 6670] 00020000:00200000:3.0:1789760772.176143:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27320320 chunk: [27320320, 27324416) 27324416, 4096 00020000:00200000:3.0:1789760772.176147:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760772.176162:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6670, 6670] 00000008:00010000:3.0:1789760772.176174:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760772.176178:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760772.176186:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27320320, 27324416) append 0 append lockpos 0 00000080:00200000:3.0:1789760772.176205:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27320320=0x1a0e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760772.176253:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760772.176255:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27324416 00000020:00010000:3.0:1789760772.176262:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760772.176265:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.176267:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.176270:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760772.176273:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760772.176279:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760772.176285:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760772.176298:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760772.176376:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107afb80 - object 0x240000400:5 offset 27320320<>27324416 00000100:00100000:2.0:1789760772.176574:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512716672:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760772.212002:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973989 gen 3 00000008:00010000:3.0:1789760772.212038:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760772.212045:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.212054:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.212075:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760772.212098:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512716672:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760772.212188:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6670, 6670] 00000080:00200000:1.0:1789760772.212209:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760772.212215:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760772.212225:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27324416 i_blocks 55303 00000020:00200000:1.0:1789760772.212228:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760772.212236:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760772.212260:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27324416, count: 4096 00000080:00200000:1.0:1789760772.212266:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27324416, bytes: 4096 00000080:00200000:1.0:1789760772.212273:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760772.212280:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760772.212282:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27324416, 27328512) 0 0 00000080:00200000:1.0:1789760772.212285:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760772.212298:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6671, 6671] 00020000:00200000:1.0:1789760772.212304:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27324416 chunk: [27324416, 27328512) 27328512, 4096 00020000:00200000:1.0:1789760772.212307:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760772.212324:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6671, 6671] 00000008:00010000:1.0:1789760772.212338:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760772.212343:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760772.212353:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27324416, 27328512) append 0 append lockpos 0 00000080:00200000:1.0:1789760772.212364:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27324416=0x1a0f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760772.212413:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760772.212416:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27328512 00000020:00010000:1.0:1789760772.212424:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760772.212427:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.212430:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.212432:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760772.212435:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760772.212441:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760772.212448:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760772.212462:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760772.212544:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96d80 - object 0x240000400:5 offset 27324416<>27328512 00000100:00100000:0.0:1789760772.215365:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760772.215394:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512716800:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760772.244334:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973990 gen 3 00000008:00010000:0.0:1789760772.244374:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760772.244381:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760772.244390:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760772.244411:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760772.244436:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512716800:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760772.244488:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6671, 6671] 00000080:00200000:1.0:1789760772.244500:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760772.244505:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760772.244517:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27328512 i_blocks 55303 00000020:00200000:1.0:1789760772.244520:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760772.244528:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760772.244553:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27328512, count: 4096 00000080:00200000:1.0:1789760772.244559:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27328512, bytes: 4096 00000080:00200000:1.0:1789760772.244565:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760772.244572:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760772.244573:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27328512, 27332608) 0 0 00000080:00200000:1.0:1789760772.244577:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760772.244589:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6672, 6672] 00020000:00200000:1.0:1789760772.244605:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27328512 chunk: [27328512, 27332608) 27332608, 4096 00020000:00200000:1.0:1789760772.244608:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760772.244623:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6672, 6672] 00000008:00010000:1.0:1789760772.244635:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760772.244640:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760772.244648:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27328512, 27332608) append 0 append lockpos 0 00000080:00200000:1.0:1789760772.244658:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27328512=0x1a10000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760772.244698:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760772.244701:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27332608 00000020:00010000:1.0:1789760772.244707:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760772.246017:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.246022:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.246025:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760772.246028:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760772.246037:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760772.246045:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760772.246338:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760772.246419:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95880 - object 0x240000400:5 offset 27328512<>27332608 00000100:00100000:0.0:1789760772.246485:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512716928:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760772.287506:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760772.287518:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760772.287526:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760772.287551:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760772.287580:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512716928:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760772.287651:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6672, 6672] 00000080:00200000:3.0:1789760772.287665:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760772.287670:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760772.287680:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27332608 i_blocks 55303 00000020:00200000:3.0:1789760772.287684:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760772.287692:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760772.287719:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27332608, count: 4096 00000080:00200000:3.0:1789760772.287725:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27332608, bytes: 4096 00000080:00200000:3.0:1789760772.287733:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760772.287740:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760772.287742:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27332608, 27336704) 0 0 00000080:00200000:3.0:1789760772.287746:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760772.287759:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6673, 6673] 00020000:00200000:3.0:1789760772.287766:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27332608 chunk: [27332608, 27336704) 27336704, 4096 00020000:00200000:3.0:1789760772.287769:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760772.287784:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6673, 6673] 00000008:00010000:3.0:1789760772.288406:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760772.288413:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760772.288423:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27332608, 27336704) append 0 append lockpos 0 00000080:00200000:3.0:1789760772.288433:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27332608=0x1a11000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760772.288481:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760772.288484:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27336704 00000020:00010000:3.0:1789760772.288490:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760772.288494:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.288497:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.288500:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760772.288503:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760772.288508:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760772.288514:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760772.288532:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760772.288694:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107afb80 - object 0x240000400:5 offset 27332608<>27336704 00000100:00100000:2.0:1789760772.288816:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512717056:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760772.324735:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973991 gen 3 00000100:00100000:3.0:1789760772.324770:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc0107afb80 x1876699512717056/t4294973992(4294973992) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760788 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760772.324880:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760772.324887:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.324896:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.325024:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760772.325044:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512717056:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760772.325136:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6673, 6673] 00000080:00200000:0.0:1789760772.325147:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.325153:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.325165:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27336704 i_blocks 55303 00000020:00200000:0.0:1789760772.325168:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.325176:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.325209:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27336704, count: 4096 00000080:00200000:0.0:1789760772.325217:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27336704, bytes: 4096 00000080:00200000:0.0:1789760772.325224:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.325231:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.325233:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27336704, 27340800) 0 0 00000080:00200000:0.0:1789760772.325237:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.325249:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6674, 6674] 00020000:00200000:0.0:1789760772.325256:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27336704 chunk: [27336704, 27340800) 27340800, 4096 00020000:00200000:0.0:1789760772.325259:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.325274:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6674, 6674] 00000008:00010000:0.0:1789760772.325288:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.325292:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760772.325301:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27336704, 27340800) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.325312:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27336704=0x1a12000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.325356:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.325359:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27340800 00000020:00010000:0.0:1789760772.325365:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.325368:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.325371:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.325374:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.325377:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.325383:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.325389:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.325403:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc600 00000008:00100000:0.0:1789760772.325477:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b800 - object 0x240000400:5 offset 27336704<>27340800 00000100:00100000:1.0:1789760772.325554:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512717184:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760772.368393:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973993 gen 3 00000008:00010000:1.0:1789760772.368440:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760772.368447:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.368456:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.368479:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760772.368507:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512717184:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760772.368609:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6674, 6674] 00000080:00200000:0.0:1789760772.368622:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.368627:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.368637:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27340800 i_blocks 55303 00000020:00200000:0.0:1789760772.368640:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.368649:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.368676:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27340800, count: 4096 00000080:00200000:0.0:1789760772.368682:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27340800, bytes: 4096 00000080:00200000:0.0:1789760772.368689:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.368696:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.368698:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27340800, 27344896) 0 0 00000080:00200000:0.0:1789760772.368701:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.368714:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6675, 6675] 00020000:00200000:0.0:1789760772.368720:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27340800 chunk: [27340800, 27344896) 27344896, 4096 00020000:00200000:0.0:1789760772.368724:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.368741:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6675, 6675] 00000008:00010000:0.0:1789760772.368753:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.368758:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760772.368766:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27340800, 27344896) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.368776:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27340800=0x1a13000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.368820:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.368823:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27344896 00000020:00010000:0.0:1789760772.368828:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.368831:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.368833:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.368836:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.368839:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.368844:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.368850:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.368864:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000008:00100000:0.0:1789760772.368971:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b480 - object 0x240000400:5 offset 27340800<>27344896 00000100:00100000:1.0:1789760772.369129:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512717312:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760772.418945:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973994 gen 3 00000008:00010000:1.0:1789760772.418980:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760772.418987:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.418996:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.419016:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760772.419039:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512717312:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760772.419454:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6675, 6675] 00000080:00200000:0.0:1789760772.419468:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.419473:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.419486:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27344896 i_blocks 55303 00000020:00200000:0.0:1789760772.419489:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.419497:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.419522:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27344896, count: 4096 00000080:00200000:0.0:1789760772.419528:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27344896, bytes: 4096 00000080:00200000:0.0:1789760772.419536:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.419543:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.419545:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27344896, 27348992) 0 0 00000080:00200000:0.0:1789760772.419549:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.419574:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6676, 6676] 00020000:00200000:0.0:1789760772.419581:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27344896 chunk: [27344896, 27348992) 27348992, 4096 00020000:00200000:0.0:1789760772.419584:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.419600:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6676, 6676] 00000008:00010000:0.0:1789760772.419614:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.419618:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760772.419627:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27344896, 27348992) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.419636:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27344896=0x1a14000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.419679:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.419682:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27348992 00000020:00010000:0.0:1789760772.419688:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.419691:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.419694:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.419697:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.419700:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.419706:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.419710:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.419725:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cccc0 00000008:00100000:0.0:1789760772.419875:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149180 - object 0x240000400:5 offset 27344896<>27348992 00000100:00100000:1.0:1789760772.421342:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760772.421370:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512717440:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760772.460478:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973995 gen 3 00000008:00010000:1.0:1789760772.460514:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760772.460520:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.460529:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.460551:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760772.460586:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512717440:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760772.460995:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6676, 6676] 00000080:00200000:0.0:1789760772.461007:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.461011:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.461022:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27348992 i_blocks 55303 00000020:00200000:0.0:1789760772.461025:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.461032:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.461054:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27348992, count: 4096 00000080:00200000:0.0:1789760772.461060:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27348992, bytes: 4096 00000080:00200000:0.0:1789760772.461068:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.461075:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.461077:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27348992, 27353088) 0 0 00000080:00200000:0.0:1789760772.461081:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.461092:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6677, 6677] 00020000:00200000:0.0:1789760772.461098:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27348992 chunk: [27348992, 27353088) 27353088, 4096 00020000:00200000:0.0:1789760772.461102:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.461115:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6677, 6677] 00000008:00010000:0.0:1789760772.461128:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.461133:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760772.461143:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27348992, 27353088) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.461153:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27348992=0x1a15000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.461199:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.461202:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27353088 00000020:00010000:0.0:1789760772.461208:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.461211:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.461214:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.461218:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.461220:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.461226:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.461231:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.461243:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc900 00000008:00100000:0.0:1789760772.461309:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148000 - object 0x240000400:5 offset 27348992<>27353088 00000100:00100000:1.0:1789760772.461361:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512717568:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760772.500773:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973996 gen 3 00000008:00010000:1.0:1789760772.500809:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760772.500815:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.500824:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.500844:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760772.500866:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512717568:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760772.502482:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6677, 6677] 00000080:00200000:0.0:1789760772.502498:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.502503:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.502517:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27353088 i_blocks 55303 00000020:00200000:0.0:1789760772.502522:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.502531:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.502557:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27353088, count: 4096 00000080:00200000:0.0:1789760772.502563:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27353088, bytes: 4096 00000080:00200000:0.0:1789760772.502574:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.502581:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.502584:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27353088, 27357184) 0 0 00000080:00200000:0.0:1789760772.502587:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.502602:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6678, 6678] 00020000:00200000:0.0:1789760772.502610:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27353088 chunk: [27353088, 27357184) 27357184, 4096 00020000:00200000:0.0:1789760772.502614:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.502632:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6678, 6678] 00000008:00010000:0.0:1789760772.502650:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.502656:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760772.502668:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27353088, 27357184) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.502679:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27353088=0x1a16000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.502730:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.502733:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27357184 00000020:00010000:0.0:1789760772.502740:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.502744:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.502746:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.502749:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.502752:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.502758:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.502774:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.502792:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000008:00100000:0.0:1789760772.502893:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149c00 - object 0x240000400:5 offset 27353088<>27357184 00000100:00100000:1.0:1789760772.503001:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512717696:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760772.546091:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973997 gen 3 00000008:00010000:1.0:1789760772.546131:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760772.546138:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.546147:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.546167:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760772.546200:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512717696:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760772.546639:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6678, 6678] 00000080:00200000:0.0:1789760772.546654:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.546659:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.546672:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27357184 i_blocks 55303 00000020:00200000:0.0:1789760772.546675:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.546683:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.546712:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27357184, count: 4096 00000080:00200000:0.0:1789760772.546719:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27357184, bytes: 4096 00000080:00200000:0.0:1789760772.546728:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.546734:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.546736:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27357184, 27361280) 0 0 00000080:00200000:0.0:1789760772.546740:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.546755:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6679, 6679] 00020000:00200000:0.0:1789760772.546762:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27357184 chunk: [27357184, 27361280) 27361280, 4096 00020000:00200000:0.0:1789760772.546766:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.546784:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6679, 6679] 00000008:00010000:0.0:1789760772.546800:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.546805:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760772.546816:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27357184, 27361280) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.546827:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27357184=0x1a17000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.546874:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.546876:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27361280 00000020:00010000:0.0:1789760772.546883:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.546887:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.546890:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.546893:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.546896:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.547691:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.547700:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.547724:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccb40 00000008:00100000:0.0:1789760772.547816:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149f80 - object 0x240000400:5 offset 27357184<>27361280 00000100:00100000:1.0:1789760772.547895:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512717824:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760772.582406:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973998 gen 3 00000008:00010000:1.0:1789760772.582445:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760772.582453:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760772.582462:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760772.582483:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760772.582506:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512717824:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760772.582608:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6679, 6679] 00000080:00200000:0.0:1789760772.582621:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760772.582626:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760772.582638:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27361280 i_blocks 55303 00000020:00200000:0.0:1789760772.582642:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760772.582649:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760772.582677:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27361280, count: 4096 00000080:00200000:0.0:1789760772.582683:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27361280, bytes: 4096 00000080:00200000:0.0:1789760772.582693:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760772.582701:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760772.582703:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27361280, 27365376) 0 0 00000080:00200000:0.0:1789760772.582707:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760772.582720:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6680, 6680] 00020000:00200000:0.0:1789760772.582727:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27361280 chunk: [27361280, 27365376) 27365376, 4096 00020000:00200000:0.0:1789760772.582731:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760772.582747:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6680, 6680] 00000008:00010000:0.0:1789760772.582762:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760772.582767:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760772.582775:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27361280, 27365376) append 0 append lockpos 0 00000080:00200000:0.0:1789760772.582785:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27361280=0x1a18000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760772.582829:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760772.582832:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27365376 00000020:00010000:0.0:1789760772.582839:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760772.582842:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.582845:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760772.582848:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760772.582850:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760772.582856:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760772.582871:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760772.582887:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc300 00000008:00100000:0.0:1789760772.584477:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149500 - object 0x240000400:5 offset 27361280<>27365376 00000100:00100000:0.0:1789760772.584548:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512717952:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760772.631136:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294973999 gen 3 00000008:00010000:0.0:1789760772.631174:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:0.0:1789760772.631181:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760772.631189:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760772.631220:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:0.0:1789760772.631244:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512717952:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760772.631405:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6680, 6680] 00000080:00200000:1.0:1789760772.631418:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760772.631422:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760772.631434:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27365376 i_blocks 55303 00000020:00200000:1.0:1789760772.631437:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760772.631444:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760772.631469:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27365376, count: 4096 00000080:00200000:1.0:1789760772.631476:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27365376, bytes: 4096 00000080:00200000:1.0:1789760772.631484:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760772.631491:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760772.631494:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27365376, 27369472) 0 0 00000080:00200000:1.0:1789760772.631497:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760772.631511:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6681, 6681] 00020000:00200000:1.0:1789760772.631517:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27365376 chunk: [27365376, 27369472) 27369472, 4096 00020000:00200000:1.0:1789760772.631521:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760772.631535:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6681, 6681] 00000008:00010000:1.0:1789760772.631548:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760772.631552:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760772.631561:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27365376, 27369472) append 0 append lockpos 0 00000080:00200000:1.0:1789760772.631572:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27365376=0x1a19000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760772.631613:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760772.631616:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27369472 00000020:00010000:1.0:1789760772.631622:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760772.631626:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.631628:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.631631:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760772.631634:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760772.631640:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760772.631645:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760772.631670:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760772.631747:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96680 - object 0x240000400:5 offset 27365376<>27369472 00000100:00100000:0.0:1789760772.631841:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512718080:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760772.677462:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974000 gen 3 00000008:00010000:0.0:1789760772.677502:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760772.677509:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760772.677586:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760772.677670:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760772.677695:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512718080:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760772.677776:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6681, 6681] 00000080:00200000:1.0:1789760772.677789:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760772.677794:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760772.677805:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27369472 i_blocks 55303 00000020:00200000:1.0:1789760772.677808:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760772.677816:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760772.677842:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27369472, count: 4096 00000080:00200000:1.0:1789760772.677848:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27369472, bytes: 4096 00000080:00200000:1.0:1789760772.677858:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760772.677865:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760772.677867:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27369472, 27373568) 0 0 00000080:00200000:1.0:1789760772.677871:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760772.677884:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6682, 6682] 00020000:00200000:1.0:1789760772.677892:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27369472 chunk: [27369472, 27373568) 27373568, 4096 00020000:00200000:1.0:1789760772.677896:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760772.677929:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6682, 6682] 00000008:00010000:1.0:1789760772.677944:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760772.677949:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760772.677957:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27369472, 27373568) append 0 append lockpos 0 00000080:00200000:1.0:1789760772.677969:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27369472=0x1a1a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760772.678012:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760772.678014:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27373568 00000020:00010000:1.0:1789760772.678021:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760772.678024:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.678027:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760772.678312:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760772.678317:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760772.678325:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760772.678332:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760772.678346:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760772.678422:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97480 - object 0x240000400:5 offset 27369472<>27373568 00000100:00100000:0.0:1789760772.678492:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512718208:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760772.734640:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974001 gen 3 00000008:00010000:0.0:1789760772.734676:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760772.734682:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760772.734691:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760772.734709:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760772.734731:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512718208:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760772.734798:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6682, 6682] 00000080:00200000:3.0:1789760772.734810:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760772.734816:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760772.734827:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27373568 i_blocks 55303 00000020:00200000:3.0:1789760772.734830:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760772.734838:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760772.734863:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27373568, count: 4096 00000080:00200000:3.0:1789760772.734869:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27373568, bytes: 4096 00000080:00200000:3.0:1789760772.734877:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760772.734884:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760772.734886:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27373568, 27377664) 0 0 00000080:00200000:3.0:1789760772.734890:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760772.734926:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6683, 6683] 00020000:00200000:3.0:1789760772.734934:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27373568 chunk: [27373568, 27377664) 27377664, 4096 00020000:00200000:3.0:1789760772.734937:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760772.734951:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6683, 6683] 00000008:00010000:3.0:1789760772.734965:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760772.734969:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760772.734979:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27373568, 27377664) append 0 append lockpos 0 00000080:00200000:3.0:1789760772.734989:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27373568=0x1a1b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760772.735033:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760772.735035:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27377664 00000020:00010000:3.0:1789760772.735042:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760772.735046:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.735049:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760772.735052:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760772.735054:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760772.735060:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760772.735065:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760772.735079:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760772.735166:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 27373568<>27377664 00000100:00100000:3.0:1789760772.735215:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760772.735228:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512718336:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760772.773532:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760772.773545:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.773554:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.773575:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760772.773606:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512718336:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760772.773925:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6683, 6683] 00000080:00200000:2.0:1789760772.773938:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760772.773943:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760772.773955:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27377664 i_blocks 55303 00000020:00200000:2.0:1789760772.773958:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760772.773967:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760772.773993:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27377664, count: 4096 00000080:00200000:2.0:1789760772.773999:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27377664, bytes: 4096 00000080:00200000:2.0:1789760772.774008:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760772.774016:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760772.774018:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27377664, 27381760) 0 0 00000080:00200000:2.0:1789760772.774021:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760772.774034:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6684, 6684] 00020000:00200000:2.0:1789760772.774041:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27377664 chunk: [27377664, 27381760) 27381760, 4096 00020000:00200000:2.0:1789760772.774045:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760772.774061:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6684, 6684] 00000008:00010000:2.0:1789760772.774074:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760772.774079:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760772.774088:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27377664, 27381760) append 0 append lockpos 0 00000080:00200000:2.0:1789760772.774100:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27377664=0x1a1c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760772.774145:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760772.774148:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27381760 00000020:00010000:2.0:1789760772.774154:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760772.774157:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.774161:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.774163:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760772.774165:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760772.774183:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760772.774198:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760772.774212:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760772.774288:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 27377664<>27381760 00000100:00100000:2.0:1789760772.774325:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760772.774339:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512718464:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760772.820920:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974003 gen 3 00000008:00010000:3.0:1789760772.820965:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760772.820972:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.820979:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.821063:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760772.821089:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512718464:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760772.821166:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6684, 6684] 00000080:00200000:2.0:1789760772.821237:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760772.821242:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760772.821253:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27381760 i_blocks 55303 00000020:00200000:2.0:1789760772.821257:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760772.821266:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760772.821325:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27381760, count: 4096 00000080:00200000:2.0:1789760772.821332:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27381760, bytes: 4096 00000080:00200000:2.0:1789760772.821344:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760772.821351:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760772.821353:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27381760, 27385856) 0 0 00000080:00200000:2.0:1789760772.821357:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760772.821370:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6685, 6685] 00020000:00200000:2.0:1789760772.821378:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27381760 chunk: [27381760, 27385856) 27385856, 4096 00020000:00200000:2.0:1789760772.821382:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760772.821396:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6685, 6685] 00000008:00010000:2.0:1789760772.821411:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760772.821416:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760772.821425:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27381760, 27385856) append 0 append lockpos 0 00000080:00200000:2.0:1789760772.821436:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27381760=0x1a1d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760772.821480:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760772.821483:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27385856 00000020:00010000:2.0:1789760772.821490:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760772.821493:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.821496:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.821499:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760772.821502:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760772.821508:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760772.821513:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760772.821527:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760772.821605:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 27381760<>27385856 00000100:00100000:3.0:1789760772.821662:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512718592:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760772.877341:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974004 gen 3 00000008:00010000:3.0:1789760772.877380:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760772.877387:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.877396:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.877414:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760772.877440:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512718592:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760772.877528:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6685, 6685] 00000080:00200000:2.0:1789760772.877539:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760772.877544:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760772.877556:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27385856 i_blocks 55303 00000020:00200000:2.0:1789760772.877559:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760772.877567:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760772.877594:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27385856, count: 4096 00000080:00200000:2.0:1789760772.877600:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27385856, bytes: 4096 00000080:00200000:2.0:1789760772.877608:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760772.877614:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760772.877616:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27385856, 27389952) 0 0 00000080:00200000:2.0:1789760772.877620:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760772.877634:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6686, 6686] 00020000:00200000:2.0:1789760772.877640:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27385856 chunk: [27385856, 27389952) 27389952, 4096 00020000:00200000:2.0:1789760772.877644:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760772.877658:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6686, 6686] 00000008:00010000:2.0:1789760772.877672:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760772.877677:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760772.877686:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27385856, 27389952) append 0 append lockpos 0 00000080:00200000:2.0:1789760772.877698:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27385856=0x1a1e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760772.877743:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760772.877758:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27389952 00000020:00010000:2.0:1789760772.877765:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760772.877769:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.877772:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.877774:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760772.877777:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760772.877784:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760772.877790:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760772.877803:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760772.877878:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 27385856<>27389952 00000100:00100000:3.0:1789760772.878108:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760772.878122:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512718720:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760772.893351:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760772.929438:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974005 gen 3 00000008:00010000:3.0:1789760772.929480:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760772.929486:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.929495:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.931086:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760772.931124:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512718720:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760772.932772:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6686, 6686] 00000080:00200000:2.0:1789760772.932787:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760772.932792:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760772.932805:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27389952 i_blocks 55303 00000020:00200000:2.0:1789760772.932810:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760772.932819:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760772.932843:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27389952, count: 4096 00000080:00200000:2.0:1789760772.932850:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27389952, bytes: 4096 00000080:00200000:2.0:1789760772.932860:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760772.932867:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760772.932868:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27389952, 27394048) 0 0 00000080:00200000:2.0:1789760772.932878:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760772.932891:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6687, 6687] 00020000:00200000:2.0:1789760772.932898:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27389952 chunk: [27389952, 27394048) 27394048, 4096 00020000:00200000:2.0:1789760772.932924:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760772.932942:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6687, 6687] 00000008:00010000:2.0:1789760772.932957:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760772.932962:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760772.932971:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27389952, 27394048) append 0 append lockpos 0 00000080:00200000:2.0:1789760772.932982:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27389952=0x1a1f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760772.933031:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760772.933034:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27394048 00000020:00010000:2.0:1789760772.933041:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760772.933044:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.933047:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.933050:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760772.933052:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760772.933059:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760772.933064:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760772.933080:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760772.933168:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 27389952<>27394048 00000100:00100000:3.0:1789760772.933536:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512718848:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760772.982294:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974006 gen 3 00000008:00010000:3.0:1789760772.982327:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760772.982334:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760772.982342:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760772.982403:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760772.982426:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512718848:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760772.982489:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6687, 6687] 00000080:00200000:2.0:1789760772.982503:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760772.982508:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760772.982520:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27394048 i_blocks 55303 00000020:00200000:2.0:1789760772.982524:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760772.982532:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760772.982558:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27394048, count: 4096 00000080:00200000:2.0:1789760772.982564:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27394048, bytes: 4096 00000080:00200000:2.0:1789760772.982573:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760772.982650:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760772.982653:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27394048, 27398144) 0 0 00000080:00200000:2.0:1789760772.982657:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760772.982740:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6688, 6688] 00020000:00200000:2.0:1789760772.982748:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27394048 chunk: [27394048, 27398144) 27398144, 4096 00020000:00200000:2.0:1789760772.982752:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760772.982767:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6688, 6688] 00000008:00010000:2.0:1789760772.982781:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760772.982785:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760772.982852:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27394048, 27398144) append 0 append lockpos 0 00000080:00200000:2.0:1789760772.982948:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27394048=0x1a20000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760772.982995:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760772.982998:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27398144 00000020:00010000:2.0:1789760772.983006:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760772.983009:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.983012:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760772.983016:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760772.983019:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760772.983025:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760772.983031:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760772.983046:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760772.983122:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 27394048<>27398144 00000100:00100000:3.0:1789760772.983185:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512718976:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760773.018493:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974007 gen 3 00000008:00010000:3.0:1789760773.018527:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760773.018533:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.018542:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.018606:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760773.018626:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512718976:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760773.018760:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6688, 6688] 00000080:00200000:2.0:1789760773.018772:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.018778:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.018798:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27398144 i_blocks 55303 00000020:00200000:2.0:1789760773.018802:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.018810:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.018832:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27398144, count: 4096 00000080:00200000:2.0:1789760773.018837:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27398144, bytes: 4096 00000080:00200000:2.0:1789760773.018844:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.018851:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.018853:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27398144, 27402240) 0 0 00000080:00200000:2.0:1789760773.018856:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.018868:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6689, 6689] 00020000:00200000:2.0:1789760773.018874:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27398144 chunk: [27398144, 27402240) 27402240, 4096 00020000:00200000:2.0:1789760773.018877:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.018890:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6689, 6689] 00000008:00010000:2.0:1789760773.018929:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.018934:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760773.018943:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27398144, 27402240) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.018955:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27398144=0x1a21000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.019001:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.019003:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27402240 00000020:00010000:2.0:1789760773.019008:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.019010:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.019012:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.019014:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.019016:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.019021:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.019025:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.019035:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760773.019097:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 27398144<>27402240 00000100:00100000:3.0:1789760773.019150:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512719104:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760773.042548:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974008 gen 3 00000008:00010000:3.0:1789760773.042594:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760773.042601:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.042609:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.042847:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760773.042873:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512719104:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760773.043167:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6689, 6689] 00000080:00200000:2.0:1789760773.043180:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.043185:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.043222:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27402240 i_blocks 55303 00000020:00200000:2.0:1789760773.043226:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.043234:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.043258:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27402240, count: 4096 00000080:00200000:2.0:1789760773.043264:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27402240, bytes: 4096 00000080:00200000:2.0:1789760773.043271:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.043278:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.043281:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27402240, 27406336) 0 0 00000080:00200000:2.0:1789760773.043285:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.043299:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6690, 6690] 00020000:00200000:2.0:1789760773.043306:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27402240 chunk: [27402240, 27406336) 27406336, 4096 00020000:00200000:2.0:1789760773.043310:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.043324:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6690, 6690] 00000008:00010000:2.0:1789760773.043338:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.043342:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760773.043351:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27402240, 27406336) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.043361:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27402240=0x1a22000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.043404:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.043407:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27406336 00000020:00010000:2.0:1789760773.043414:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.043417:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.043419:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.043422:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.043424:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.043431:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.043436:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.043448:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760773.043523:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 27402240<>27406336 00000100:00100000:3.0:1789760773.043642:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512719232:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760773.083253:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974009 gen 3 00000008:00010000:3.0:1789760773.083425:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760773.083432:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.083440:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.083501:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760773.083527:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512719232:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760773.083586:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6690, 6690] 00000080:00200000:2.0:1789760773.083599:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.083604:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.083615:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27406336 i_blocks 55303 00000020:00200000:2.0:1789760773.083618:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.083627:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.083653:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27406336, count: 4096 00000080:00200000:2.0:1789760773.083659:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27406336, bytes: 4096 00000080:00200000:2.0:1789760773.083666:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.083682:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.083684:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27406336, 27410432) 0 0 00000080:00200000:2.0:1789760773.083688:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.083701:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6691, 6691] 00020000:00200000:2.0:1789760773.083708:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27406336 chunk: [27406336, 27410432) 27410432, 4096 00020000:00200000:2.0:1789760773.083712:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.083725:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6691, 6691] 00000008:00010000:2.0:1789760773.083781:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.083796:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760773.083808:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27406336, 27410432) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.083818:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27406336=0x1a23000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.083863:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.083866:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27410432 00000020:00010000:2.0:1789760773.083873:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.083876:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.083879:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.083884:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.083887:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.083893:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.084080:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.084094:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760773.084165:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 27406336<>27410432 00000100:00100000:3.0:1789760773.084260:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512719360:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760773.125084:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760773.125095:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.125103:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.125150:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760773.125172:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512719360:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760773.125235:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6691, 6691] 00000080:00200000:2.0:1789760773.125247:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.125252:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.125263:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27410432 i_blocks 55303 00000020:00200000:2.0:1789760773.125267:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.125274:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.125298:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27410432, count: 4096 00000080:00200000:2.0:1789760773.125304:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27410432, bytes: 4096 00000080:00200000:2.0:1789760773.125311:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.125318:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.125320:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27410432, 27414528) 0 0 00000080:00200000:2.0:1789760773.125323:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.125336:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6692, 6692] 00020000:00200000:2.0:1789760773.125342:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27410432 chunk: [27410432, 27414528) 27414528, 4096 00020000:00200000:2.0:1789760773.125346:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.125360:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6692, 6692] 00000008:00010000:2.0:1789760773.125375:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.125379:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760773.125389:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27410432, 27414528) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.125398:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27410432=0x1a24000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.125440:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.125442:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27414528 00000020:00010000:2.0:1789760773.125448:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.125451:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.125454:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.125457:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.125459:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.125466:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.125471:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.125483:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000008:00100000:2.0:1789760773.125556:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 27410432<>27414528 00000100:00100000:3.0:1789760773.125607:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512719488:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760773.162504:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974011 gen 3 00000008:00010000:3.0:1789760773.162552:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000020:00200000:3.0:1789760773.162559:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.162567:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.162636:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d3c0 00000100:00100000:3.0:1789760773.162662:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512719488:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760773.162821:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6692, 6692] 00000080:00200000:2.0:1789760773.162836:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.162841:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.162853:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27414528 i_blocks 55303 00000020:00200000:2.0:1789760773.162865:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.162874:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.162973:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27414528, count: 4096 00000080:00200000:2.0:1789760773.162981:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27414528, bytes: 4096 00000080:00200000:2.0:1789760773.162990:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.162996:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.162999:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27414528, 27418624) 0 0 00000080:00200000:2.0:1789760773.163003:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.163018:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6693, 6693] 00020000:00200000:2.0:1789760773.163025:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27414528 chunk: [27414528, 27418624) 27418624, 4096 00020000:00200000:2.0:1789760773.163029:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.163045:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6693, 6693] 00000008:00010000:2.0:1789760773.163061:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.163065:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760773.163075:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27414528, 27418624) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.163086:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27414528=0x1a25000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.163131:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.163134:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27418624 00000020:00010000:2.0:1789760773.163141:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.163144:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.163147:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.163150:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.163153:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.163158:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.163164:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.163178:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760773.163263:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 27414528<>27418624 00000100:00100000:3.0:1789760773.163456:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512719616:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760773.196541:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974012 gen 3 00000008:00010000:3.0:1789760773.196574:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760773.196580:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.196649:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.196770:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760773.196804:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512719616:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760773.196998:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6693, 6693] 00000080:00200000:2.0:1789760773.197010:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.197015:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.197026:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27418624 i_blocks 55303 00000020:00200000:2.0:1789760773.197029:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.197037:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.197062:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27418624, count: 4096 00000080:00200000:2.0:1789760773.197068:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27418624, bytes: 4096 00000080:00200000:2.0:1789760773.197076:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.197082:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.197085:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27418624, 27422720) 0 0 00000080:00200000:2.0:1789760773.197088:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.197100:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6694, 6694] 00020000:00200000:2.0:1789760773.197106:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27418624 chunk: [27418624, 27422720) 27422720, 4096 00020000:00200000:2.0:1789760773.197110:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.197125:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6694, 6694] 00000008:00010000:2.0:1789760773.197139:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.197143:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760773.197154:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27418624, 27422720) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.197166:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27418624=0x1a26000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.197215:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.197218:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27422720 00000020:00010000:2.0:1789760773.197225:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.197228:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.197231:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.197234:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.197237:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.197243:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.197248:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.197262:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760773.197404:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 27418624<>27422720 00000100:00100000:3.0:1789760773.197462:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512719744:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760773.240741:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974013 gen 3 00000008:00010000:3.0:1789760773.240777:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760773.240783:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.240791:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.240810:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760773.240834:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512719744:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760773.241012:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6694, 6694] 00000080:00200000:2.0:1789760773.241024:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.241029:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.241045:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27422720 i_blocks 55303 00000020:00200000:2.0:1789760773.241049:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.241150:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.241177:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27422720, count: 4096 00000080:00200000:2.0:1789760773.241183:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27422720, bytes: 4096 00000080:00200000:2.0:1789760773.241200:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.241207:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.241209:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27422720, 27426816) 0 0 00000080:00200000:2.0:1789760773.241212:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.241226:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6695, 6695] 00020000:00200000:2.0:1789760773.241232:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27422720 chunk: [27422720, 27426816) 27426816, 4096 00020000:00200000:2.0:1789760773.241236:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.241250:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6695, 6695] 00000008:00010000:2.0:1789760773.241262:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.241266:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760773.241275:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27422720, 27426816) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.241285:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27422720=0x1a27000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.241331:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.241333:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27426816 00000020:00010000:2.0:1789760773.241339:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.241343:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.241346:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.241349:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.241352:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.241358:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.241364:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.241378:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000008:00100000:2.0:1789760773.241460:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 27422720<>27426816 00000100:00100000:3.0:1789760773.241523:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512719872:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760773.272589:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974014 gen 3 00000008:00010000:3.0:1789760773.272622:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760773.272628:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.272635:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.272651:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760773.272678:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512719872:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760773.272807:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6695, 6695] 00000080:00200000:2.0:1789760773.272819:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.272824:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.272835:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27426816 i_blocks 55303 00000020:00200000:2.0:1789760773.272838:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.272845:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.272869:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27426816, count: 4096 00000080:00200000:2.0:1789760773.272874:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27426816, bytes: 4096 00000080:00200000:2.0:1789760773.272881:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.272888:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.272890:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27426816, 27430912) 0 0 00000080:00200000:2.0:1789760773.272893:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.272930:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6696, 6696] 00020000:00200000:2.0:1789760773.272937:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27426816 chunk: [27426816, 27430912) 27430912, 4096 00020000:00200000:2.0:1789760773.272941:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.272956:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6696, 6696] 00000008:00010000:2.0:1789760773.272970:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.272975:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760773.272984:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27426816, 27430912) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.272994:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27426816=0x1a28000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.273039:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.273042:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27430912 00000020:00010000:2.0:1789760773.273049:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.273052:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.273055:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.273058:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.273060:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.273066:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.273071:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.273085:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000008:00100000:2.0:1789760773.273156:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 27426816<>27430912 00000100:00100000:3.0:1789760773.273251:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760773.273265:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512720000:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760773.304156:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974015 gen 3 00000008:00010000:3.0:1789760773.304219:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000020:00200000:3.0:1789760773.304234:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.304246:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.304264:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d0c0 00000100:00100000:3.0:1789760773.304296:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512720000:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760773.304451:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6696, 6696] 00000080:00200000:2.0:1789760773.304467:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.304474:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.304491:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27430912 i_blocks 55303 00000020:00200000:2.0:1789760773.304495:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.304504:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.304545:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27430912, count: 4096 00000080:00200000:2.0:1789760773.304553:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27430912, bytes: 4096 00000080:00200000:2.0:1789760773.304562:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.304571:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.304573:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27430912, 27435008) 0 0 00000080:00200000:2.0:1789760773.304586:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.304604:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6697, 6697] 00020000:00200000:2.0:1789760773.304612:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27430912 chunk: [27430912, 27435008) 27435008, 4096 00020000:00200000:2.0:1789760773.304617:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.304638:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6697, 6697] 00000008:00010000:2.0:1789760773.304667:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.304673:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760773.304686:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27430912, 27435008) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.304701:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27430912=0x1a29000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.304759:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.304762:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27435008 00000020:00010000:2.0:1789760773.304771:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.304775:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.304778:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.304781:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.304784:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.304792:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.304798:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.304814:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760773.304967:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 27430912<>27435008 00000100:00100000:3.0:1789760773.305173:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512720128:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760773.348150:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974016 gen 3 00000008:00010000:3.0:1789760773.348210:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760773.348219:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.348231:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.348252:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760773.348282:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512720128:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760773.349403:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6697, 6697] 00000080:00200000:2.0:1789760773.349425:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.349432:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.349450:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27435008 i_blocks 55303 00000020:00200000:2.0:1789760773.349455:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.349466:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.349502:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27435008, count: 4096 00000080:00200000:2.0:1789760773.349510:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27435008, bytes: 4096 00000080:00200000:2.0:1789760773.349522:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.349539:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.349541:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27435008, 27439104) 0 0 00000080:00200000:2.0:1789760773.349546:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.349563:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6698, 6698] 00020000:00200000:2.0:1789760773.349572:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27435008 chunk: [27435008, 27439104) 27439104, 4096 00020000:00200000:2.0:1789760773.349577:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.349599:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6698, 6698] 00000008:00010000:2.0:1789760773.349619:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.349625:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760773.349637:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27435008, 27439104) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.349651:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27435008=0x1a2a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.349717:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.349720:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27439104 00000020:00010000:2.0:1789760773.349727:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.349730:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.349733:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc00877c008/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.349737:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.349739:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.349747:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.349754:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.349775:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760773.349883:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 27435008<>27439104 00000100:00100000:3.0:1789760773.350207:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760773.350233:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512720256:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760773.396731:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974017 gen 3 00000008:00010000:2.0:1789760773.397435:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:2.0:1789760773.397445:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760773.397457:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760773.397526:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:2.0:1789760773.397567:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512720256:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760773.397760:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6698, 6698] 00000080:00200000:3.0:1789760773.397776:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760773.397782:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.398022:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27439104 i_blocks 55303 00000020:00200000:2.0:1789760773.398028:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.398038:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.398075:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27439104, count: 4096 00000080:00200000:2.0:1789760773.398083:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27439104, bytes: 4096 00000080:00200000:2.0:1789760773.398097:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.398105:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.398108:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27439104, 27443200) 0 0 00000080:00200000:2.0:1789760773.398113:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.398130:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6699, 6699] 00020000:00200000:2.0:1789760773.398139:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27439104 chunk: [27439104, 27443200) 27443200, 4096 00020000:00200000:2.0:1789760773.398143:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.398163:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6699, 6699] 00000008:00010000:2.0:1789760773.398182:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.399485:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760773.399507:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27439104, 27443200) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.399524:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27439104=0x1a2b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.399764:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.399767:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27443200 00000020:00010000:2.0:1789760773.399777:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.399781:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.399784:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.399787:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.399789:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.399799:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.399807:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.400023:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760773.400132:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 27439104<>27443200 00000100:00100000:3.0:1789760773.400277:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512720384:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760773.439819:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760773.439835:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.439846:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.439871:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760773.440283:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512720384:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760773.440460:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6699, 6699] 00000080:00200000:2.0:1789760773.440476:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.440482:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.440497:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27443200 i_blocks 55303 00000020:00200000:2.0:1789760773.440505:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.440516:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.440549:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27443200, count: 4096 00000080:00200000:2.0:1789760773.440561:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27443200, bytes: 4096 00000080:00200000:2.0:1789760773.440574:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.440816:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.440818:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27443200, 27447296) 0 0 00000080:00200000:2.0:1789760773.440822:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.440840:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6700, 6700] 00020000:00200000:2.0:1789760773.440849:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27443200 chunk: [27443200, 27447296) 27447296, 4096 00020000:00200000:2.0:1789760773.440854:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.440874:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6700, 6700] 00000008:00010000:2.0:1789760773.440892:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.440923:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760773.440936:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27443200, 27447296) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.440951:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27443200=0x1a2c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.441016:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.441019:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27447296 00000020:00010000:2.0:1789760773.441031:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.441035:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.441038:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.441041:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.441044:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.441053:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.441059:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.441078:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760773.441212:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 27443200<>27447296 00000100:00100000:3.0:1789760773.443085:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760773.443121:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512720512:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760773.504775:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974018 gen 3 00000100:00100000:3.0:1789760773.504816:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368f800 x1876699512720512/t4294974019(4294974019) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760789 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760773.504875:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760773.504883:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.504893:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.504949:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760773.504973:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512720512:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760773.505249:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6700, 6700] 00000080:00200000:2.0:1789760773.505266:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.505272:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.505286:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27447296 i_blocks 55303 00000020:00200000:2.0:1789760773.505289:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.505298:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.505331:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27447296, count: 4096 00000080:00200000:2.0:1789760773.505338:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27447296, bytes: 4096 00000080:00200000:2.0:1789760773.505348:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.505356:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.505359:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27447296, 27451392) 0 0 00000080:00200000:2.0:1789760773.505363:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.505381:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6701, 6701] 00020000:00200000:2.0:1789760773.505389:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27447296 chunk: [27447296, 27451392) 27451392, 4096 00020000:00200000:2.0:1789760773.505393:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.505413:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6701, 6701] 00000008:00010000:2.0:1789760773.505432:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.505439:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760773.505451:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27447296, 27451392) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.505464:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27447296=0x1a2d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.505521:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.505523:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27451392 00000020:00010000:2.0:1789760773.505532:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.505536:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.505539:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.505542:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.505545:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.505554:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.505561:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.505754:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760773.505967:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 27447296<>27451392 00000100:00100000:3.0:1789760773.506092:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512720640:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760773.587634:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974020 gen 3 00000008:00010000:3.0:1789760773.587704:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760773.587713:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.587724:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.587747:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760773.587780:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512720640:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760773.587976:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6701, 6701] 00000080:00200000:2.0:1789760773.587994:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.588001:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.588016:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27451392 i_blocks 55303 00000020:00200000:2.0:1789760773.588020:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.588030:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.588071:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27451392, count: 4096 00000080:00200000:2.0:1789760773.588079:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27451392, bytes: 4096 00000080:00200000:2.0:1789760773.588089:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.588098:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.588100:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27451392, 27455488) 0 0 00000080:00200000:2.0:1789760773.588106:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.588123:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6702, 6702] 00020000:00200000:2.0:1789760773.588132:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27451392 chunk: [27451392, 27455488) 27455488, 4096 00020000:00200000:2.0:1789760773.588137:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.588156:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6702, 6702] 00000008:00010000:2.0:1789760773.588175:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.588181:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760773.588203:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27451392, 27455488) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.588217:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27451392=0x1a2e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.588275:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.588278:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27455488 00000020:00010000:2.0:1789760773.588286:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.588290:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.588293:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.588297:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.588300:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.588308:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.588394:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.588411:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760773.588524:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 27451392<>27455488 00000100:00100000:3.0:1789760773.588933:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760773.588955:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512720768:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760773.658674:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760773.658689:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.658701:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.658728:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760773.658764:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512720768:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760773.659100:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6702, 6702] 00000080:00200000:2.0:1789760773.659118:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.659124:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.659148:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27455488 i_blocks 55303 00000020:00200000:2.0:1789760773.659152:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.659162:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.659206:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27455488, count: 4096 00000080:00200000:2.0:1789760773.659214:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27455488, bytes: 4096 00000080:00200000:2.0:1789760773.659229:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.659237:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.659239:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27455488, 27459584) 0 0 00000080:00200000:2.0:1789760773.659244:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.659261:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6703, 6703] 00020000:00200000:2.0:1789760773.659269:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27455488 chunk: [27455488, 27459584) 27459584, 4096 00020000:00200000:2.0:1789760773.659272:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.659291:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6703, 6703] 00000008:00010000:2.0:1789760773.659309:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.659315:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760773.659328:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27455488, 27459584) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.659342:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27455488=0x1a2f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.659399:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.659402:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27459584 00000020:00010000:2.0:1789760773.659421:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.659425:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.659428:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.659432:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.659436:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.659443:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.659450:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.659467:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760773.659585:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 27455488<>27459584 00000100:00100000:3.0:1789760773.659838:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512720896:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760773.718977:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974021 gen 3 00000100:00100000:3.0:1789760773.719017:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368e680 x1876699512720896/t4294974022(4294974022) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760789 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760773.719064:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760773.719071:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.719080:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.719097:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760773.719119:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512720896:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760773.719262:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6703, 6703] 00000080:00200000:2.0:1789760773.719278:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.719284:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.719298:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27459584 i_blocks 55303 00000020:00200000:2.0:1789760773.719302:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.719311:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.719345:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27459584, count: 4096 00000080:00200000:2.0:1789760773.719352:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27459584, bytes: 4096 00000080:00200000:2.0:1789760773.719361:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.719369:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.719372:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27459584, 27463680) 0 0 00000080:00200000:2.0:1789760773.719376:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.719392:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6704, 6704] 00020000:00200000:2.0:1789760773.719400:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27459584 chunk: [27459584, 27463680) 27463680, 4096 00020000:00200000:2.0:1789760773.719405:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.719432:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6704, 6704] 00000008:00010000:2.0:1789760773.719453:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.719458:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760773.719470:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27459584, 27463680) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.719488:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27459584=0x1a30000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.719548:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.719551:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27463680 00000020:00010000:2.0:1789760773.719559:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.719563:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.719566:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.719569:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.719573:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.719581:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.719588:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.719606:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760773.719733:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 27459584<>27463680 00000100:00100000:3.0:1789760773.719956:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760773.719976:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512721024:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760773.782662:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974023 gen 3 00000008:00010000:3.0:1789760773.782725:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760773.782735:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.782745:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.782765:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760773.782797:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512721024:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760773.782927:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6704, 6704] 00000080:00200000:2.0:1789760773.782943:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.782949:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.782964:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27463680 i_blocks 55303 00000020:00200000:2.0:1789760773.782967:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.782975:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.783009:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27463680, count: 4096 00000080:00200000:2.0:1789760773.783015:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27463680, bytes: 4096 00000080:00200000:2.0:1789760773.783024:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.783032:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.783034:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27463680, 27467776) 0 0 00000080:00200000:2.0:1789760773.783038:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.783056:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6705, 6705] 00020000:00200000:2.0:1789760773.783065:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27463680 chunk: [27463680, 27467776) 27467776, 4096 00020000:00200000:2.0:1789760773.783070:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.783092:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6705, 6705] 00000008:00010000:2.0:1789760773.783113:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.783120:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760773.783145:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27463680, 27467776) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.783160:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27463680=0x1a31000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.783223:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.783226:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27467776 00000020:00010000:2.0:1789760773.783234:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.783238:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.783241:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.783244:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.783247:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.783254:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.783261:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.783276:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760773.783375:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 27463680<>27467776 00000100:00100000:3.0:1789760773.783613:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512721152:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760773.824535:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974024 gen 3 00000008:00010000:3.0:1789760773.824600:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760773.824609:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.824620:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.824745:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760773.824781:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512721152:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760773.824879:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6705, 6705] 00000080:00200000:2.0:1789760773.824898:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.824935:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.824952:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27467776 i_blocks 55303 00000020:00200000:2.0:1789760773.824957:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.824967:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.825002:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27467776, count: 4096 00000080:00200000:2.0:1789760773.825009:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27467776, bytes: 4096 00000080:00200000:2.0:1789760773.825020:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.825029:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.825032:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27467776, 27471872) 0 0 00000080:00200000:2.0:1789760773.825036:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.825059:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6706, 6706] 00020000:00200000:2.0:1789760773.825069:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27467776 chunk: [27467776, 27471872) 27471872, 4096 00020000:00200000:2.0:1789760773.825073:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.825095:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6706, 6706] 00000008:00010000:2.0:1789760773.825112:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.825118:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760773.825130:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27467776, 27471872) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.825144:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27467776=0x1a32000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.826632:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.826641:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27471872 00000020:00010000:2.0:1789760773.826653:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.826658:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.826662:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.826667:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.826670:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.826680:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.826690:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.826720:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760773.826835:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 27467776<>27471872 00000100:00100000:3.0:1789760773.827823:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512721280:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760773.867526:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760773.867540:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.867550:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.867667:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760773.867700:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512721280:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760773.867779:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6706, 6706] 00000080:00200000:2.0:1789760773.868059:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.868067:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.868081:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27471872 i_blocks 55303 00000020:00200000:2.0:1789760773.868084:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.868093:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.868127:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27471872, count: 4096 00000080:00200000:2.0:1789760773.868133:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27471872, bytes: 4096 00000080:00200000:2.0:1789760773.868144:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.868153:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.868155:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27471872, 27475968) 0 0 00000080:00200000:2.0:1789760773.868160:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.868369:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6707, 6707] 00020000:00200000:2.0:1789760773.868380:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27471872 chunk: [27471872, 27475968) 27475968, 4096 00020000:00200000:2.0:1789760773.868384:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.868406:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6707, 6707] 00000008:00010000:2.0:1789760773.868425:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.868432:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760773.868444:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27471872, 27475968) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.868458:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27471872=0x1a33000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.868525:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.868528:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27475968 00000020:00010000:2.0:1789760773.868537:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.868540:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.868543:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.868546:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.868549:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.868557:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.868563:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.868652:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760773.868761:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 27471872<>27475968 00000100:00100000:3.0:1789760773.868859:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512721408:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760773.917055:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760773.917331:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974026 gen 3 00000008:00010000:3.0:1789760773.917392:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760773.917400:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.917411:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.917429:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760773.917456:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512721408:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760773.917685:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6707, 6707] 00000080:00200000:2.0:1789760773.917702:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.917707:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.917723:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27475968 i_blocks 55303 00000020:00200000:2.0:1789760773.917727:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.917737:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.917775:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27475968, count: 4096 00000080:00200000:2.0:1789760773.917783:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27475968, bytes: 4096 00000080:00200000:2.0:1789760773.918015:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.918025:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.918028:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27475968, 27480064) 0 0 00000080:00200000:2.0:1789760773.918033:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.918050:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6708, 6708] 00020000:00200000:2.0:1789760773.918058:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27475968 chunk: [27475968, 27480064) 27480064, 4096 00020000:00200000:2.0:1789760773.918063:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.918085:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6708, 6708] 00000008:00010000:2.0:1789760773.918115:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.918121:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760773.918134:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27475968, 27480064) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.918147:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27475968=0x1a34000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.918218:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.918222:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27480064 00000020:00010000:2.0:1789760773.918230:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.918235:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.918238:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.918241:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.918244:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.918252:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.918260:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.918279:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760773.918387:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 27475968<>27480064 00000100:00100000:3.0:1789760773.918551:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512721536:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760773.970039:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974027 gen 3 00000008:00010000:3.0:1789760773.970087:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760773.970095:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760773.970106:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760773.970174:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760773.970217:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512721536:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760773.970308:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6708, 6708] 00000080:00200000:2.0:1789760773.970325:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760773.970331:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760773.970417:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27480064 i_blocks 55303 00000020:00200000:2.0:1789760773.970422:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760773.970433:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760773.970468:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27480064, count: 4096 00000080:00200000:2.0:1789760773.970476:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27480064, bytes: 4096 00000080:00200000:2.0:1789760773.970486:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760773.970495:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760773.970497:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27480064, 27484160) 0 0 00000080:00200000:2.0:1789760773.970502:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760773.970519:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6709, 6709] 00020000:00200000:2.0:1789760773.970527:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27480064 chunk: [27480064, 27484160) 27484160, 4096 00020000:00200000:2.0:1789760773.970532:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760773.970550:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6709, 6709] 00000008:00010000:2.0:1789760773.970568:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760773.970642:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760773.970656:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27480064, 27484160) append 0 append lockpos 0 00000080:00200000:2.0:1789760773.970671:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27480064=0x1a35000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760773.970734:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760773.970802:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27484160 00000020:00010000:2.0:1789760773.970812:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760773.970816:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.970819:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760773.970823:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760773.970826:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760773.970833:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760773.970840:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760773.971074:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760773.971177:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 27480064<>27484160 00000100:00100000:2.0:1789760773.971304:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760773.971325:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512721664:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760774.021581:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974028 gen 3 00000008:00010000:3.0:1789760774.021634:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760774.021643:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.021656:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.021682:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760774.021713:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512721664:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760774.022054:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6709, 6709] 00000080:00200000:2.0:1789760774.022073:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760774.022079:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760774.022094:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27484160 i_blocks 55303 00000020:00200000:2.0:1789760774.022097:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760774.022108:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760774.022145:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27484160, count: 4096 00000080:00200000:2.0:1789760774.022152:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27484160, bytes: 4096 00000080:00200000:2.0:1789760774.022163:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760774.022173:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760774.022175:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27484160, 27488256) 0 0 00000080:00200000:2.0:1789760774.022311:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760774.022330:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6710, 6710] 00020000:00200000:2.0:1789760774.022339:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27484160 chunk: [27484160, 27488256) 27488256, 4096 00020000:00200000:2.0:1789760774.022344:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760774.022365:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6710, 6710] 00000008:00010000:2.0:1789760774.022385:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760774.022390:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760774.022402:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27484160, 27488256) append 0 append lockpos 0 00000080:00200000:2.0:1789760774.022420:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27484160=0x1a36000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760774.022489:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760774.022492:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27488256 00000020:00010000:2.0:1789760774.022499:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760774.022502:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760774.022505:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760774.022509:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760774.022512:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760774.022522:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760774.022529:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760774.022547:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760774.022661:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 27484160<>27488256 00000100:00100000:3.0:1789760774.022775:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512721792:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760774.057107:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974029 gen 3 00000008:00010000:3.0:1789760774.057175:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760774.057194:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760774.057205:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760774.057221:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760774.057249:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512721792:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760774.057531:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6710, 6710] 00000080:00200000:0.0:1789760774.057549:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.057558:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.057577:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27488256 i_blocks 55303 00000020:00200000:0.0:1789760774.057580:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.057591:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.057631:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27488256, count: 4096 00000080:00200000:0.0:1789760774.057639:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27488256, bytes: 4096 00000080:00200000:0.0:1789760774.057651:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.057659:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.057662:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27488256, 27492352) 0 0 00000080:00200000:0.0:1789760774.057666:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.057684:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6711, 6711] 00020000:00200000:0.0:1789760774.057692:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27488256 chunk: [27488256, 27492352) 27492352, 4096 00020000:00200000:0.0:1789760774.057697:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.057717:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6711, 6711] 00000008:00010000:0.0:1789760774.057735:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.057741:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760774.057754:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27488256, 27492352) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.057769:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27488256=0x1a37000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.057832:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.057835:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27492352 00000020:00010000:0.0:1789760774.057844:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.057847:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.057850:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.057854:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.057857:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.057865:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.057872:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.057890:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc300 00000008:00100000:0.0:1789760774.058050:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149500 - object 0x240000400:5 offset 27488256<>27492352 00000100:00100000:0.0:1789760774.058123:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760774.058142:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512721920:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760774.104522:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974030 gen 3 00000008:00010000:0.0:1789760774.104573:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:0.0:1789760774.104582:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.104594:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.104732:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:0.0:1789760774.104760:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512721920:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760774.105102:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6711, 6711] 00000080:00200000:1.0:1789760774.105129:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.105135:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.105152:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27492352 i_blocks 55303 00000020:00200000:1.0:1789760774.105156:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.105632:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.105669:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27492352, count: 4096 00000080:00200000:1.0:1789760774.105677:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27492352, bytes: 4096 00000080:00200000:1.0:1789760774.105689:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.105698:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.105700:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27492352, 27496448) 0 0 00000080:00200000:1.0:1789760774.105705:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.105720:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6712, 6712] 00020000:00200000:1.0:1789760774.105729:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27492352 chunk: [27492352, 27496448) 27496448, 4096 00020000:00200000:1.0:1789760774.105734:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.105753:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6712, 6712] 00000008:00010000:1.0:1789760774.105769:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.105775:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760774.105795:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27492352, 27496448) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.105811:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27492352=0x1a38000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.106033:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.106037:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27496448 00000020:00010000:1.0:1789760774.106046:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.106050:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.106053:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.106056:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.106059:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.106069:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.106076:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.106100:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760774.106337:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 27492352<>27496448 00000100:00100000:0.0:1789760774.106425:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512722048:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760774.150706:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974031 gen 3 00000008:00010000:0.0:1789760774.150762:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760774.150770:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.150781:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.150805:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760774.150835:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512722048:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760774.151059:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6712, 6712] 00000080:00200000:1.0:1789760774.151075:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.151081:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.151096:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27496448 i_blocks 55303 00000020:00200000:1.0:1789760774.151099:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.151108:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.151141:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27496448, count: 4096 00000080:00200000:1.0:1789760774.151146:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27496448, bytes: 4096 00000080:00200000:1.0:1789760774.151153:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.151161:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.151163:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27496448, 27500544) 0 0 00000080:00200000:1.0:1789760774.151167:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.151195:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6713, 6713] 00020000:00200000:1.0:1789760774.151204:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27496448 chunk: [27496448, 27500544) 27500544, 4096 00020000:00200000:1.0:1789760774.151208:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.151288:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6713, 6713] 00000008:00010000:1.0:1789760774.151306:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.151312:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760774.151325:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27496448, 27500544) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.151340:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27496448=0x1a39000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.151397:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.151399:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27500544 00000020:00010000:1.0:1789760774.151408:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.151412:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.151415:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.151418:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.151421:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.151430:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.151436:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.151453:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760774.151552:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 27496448<>27500544 00000100:00100000:0.0:1789760774.151678:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512722176:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760774.179168:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974032 gen 3 00000008:00010000:0.0:1789760774.179340:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760774.179349:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.179360:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.179389:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760774.179424:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512722176:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760774.179565:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6713, 6713] 00000080:00200000:1.0:1789760774.179581:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.179587:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.179602:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27500544 i_blocks 55303 00000020:00200000:1.0:1789760774.179606:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.179626:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.179662:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27500544, count: 4096 00000080:00200000:1.0:1789760774.179671:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27500544, bytes: 4096 00000080:00200000:1.0:1789760774.179681:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.179690:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.179692:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27500544, 27504640) 0 0 00000080:00200000:1.0:1789760774.179696:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.179714:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6714, 6714] 00020000:00200000:1.0:1789760774.179724:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27500544 chunk: [27500544, 27504640) 27504640, 4096 00020000:00200000:1.0:1789760774.179728:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.179747:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6714, 6714] 00000008:00010000:1.0:1789760774.179765:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.179771:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760774.179782:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27500544, 27504640) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.179797:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27500544=0x1a3a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.179862:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.179865:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27504640 00000020:00010000:1.0:1789760774.179873:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.179877:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.179880:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.179884:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.179887:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.179894:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.179938:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.179954:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760774.180055:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 27500544<>27504640 00000100:00100000:0.0:1789760774.180153:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760774.180170:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512722304:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760774.235582:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974033 gen 3 00000008:00010000:0.0:1789760774.235632:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760774.235640:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.235653:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.235676:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760774.235707:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512722304:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760774.235858:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6714, 6714] 00000080:00200000:1.0:1789760774.235874:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.235881:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.235896:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27504640 i_blocks 55303 00000020:00200000:1.0:1789760774.235933:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.235943:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.235976:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27504640, count: 4096 00000080:00200000:1.0:1789760774.235983:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27504640, bytes: 4096 00000080:00200000:1.0:1789760774.235994:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.236002:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.236004:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27504640, 27508736) 0 0 00000080:00200000:1.0:1789760774.236009:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.236025:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6715, 6715] 00020000:00200000:1.0:1789760774.236034:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27504640 chunk: [27504640, 27508736) 27508736, 4096 00020000:00200000:1.0:1789760774.236038:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.236058:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6715, 6715] 00000008:00010000:1.0:1789760774.236076:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.236083:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760774.236094:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27504640, 27508736) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.236121:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27504640=0x1a3b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.236189:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.236192:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27508736 00000020:00010000:1.0:1789760774.236200:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.236204:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.236207:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.236210:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.236214:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.236221:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.236228:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.236243:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760774.236344:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94380 - object 0x240000400:5 offset 27504640<>27508736 00000100:00100000:0.0:1789760774.236501:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512722432:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760774.281062:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760774.281079:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.281090:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.281112:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760774.281144:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512722432:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760774.281484:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6715, 6715] 00000080:00200000:1.0:1789760774.281502:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.281508:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.281523:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27508736 i_blocks 55303 00000020:00200000:1.0:1789760774.281527:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.281536:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.281735:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27508736, count: 4096 00000080:00200000:1.0:1789760774.281744:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27508736, bytes: 4096 00000080:00200000:1.0:1789760774.281755:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.281764:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.281767:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27508736, 27512832) 0 0 00000080:00200000:1.0:1789760774.281772:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.282081:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6716, 6716] 00020000:00200000:1.0:1789760774.282092:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27508736 chunk: [27508736, 27512832) 27512832, 4096 00020000:00200000:1.0:1789760774.282097:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.282120:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6716, 6716] 00000008:00010000:1.0:1789760774.282138:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.282144:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760774.282156:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27508736, 27512832) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.282170:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27508736=0x1a3c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.282244:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.282247:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27512832 00000020:00010000:1.0:1789760774.282254:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.282257:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.282260:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.282263:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.282266:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.282274:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.282282:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.282303:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760774.282470:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 27508736<>27512832 00000100:00100000:0.0:1789760774.282672:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760774.282696:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512722560:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760774.338292:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974035 gen 3 00000008:00010000:0.0:1789760774.338360:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760774.338369:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.338381:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.338406:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760774.338440:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512722560:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760774.338560:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6716, 6716] 00000080:00200000:1.0:1789760774.338578:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.338584:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.338599:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27512832 i_blocks 55303 00000020:00200000:1.0:1789760774.338603:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.338613:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.338649:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27512832, count: 4096 00000080:00200000:1.0:1789760774.338656:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27512832, bytes: 4096 00000080:00200000:1.0:1789760774.338666:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.338677:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.338679:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27512832, 27516928) 0 0 00000080:00200000:1.0:1789760774.338684:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.338701:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6717, 6717] 00020000:00200000:1.0:1789760774.338711:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27512832 chunk: [27512832, 27516928) 27516928, 4096 00020000:00200000:1.0:1789760774.338716:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.338737:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6717, 6717] 00000008:00010000:1.0:1789760774.338754:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.338760:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760774.338771:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27512832, 27516928) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.338787:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27512832=0x1a3d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.338845:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.338848:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27516928 00000020:00010000:1.0:1789760774.338856:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.338860:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.338863:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.338866:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.338869:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.338876:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.338884:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.339022:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760774.339203:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94700 - object 0x240000400:5 offset 27512832<>27516928 00000100:00100000:0.0:1789760774.339261:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512722688:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760774.375323:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974036 gen 3 00000008:00010000:0.0:1789760774.375386:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760774.375397:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.375412:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.375450:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760774.375488:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512722688:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760774.375659:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6717, 6717] 00000080:00200000:2.0:1789760774.375680:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760774.375686:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760774.375706:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27516928 i_blocks 55303 00000020:00200000:2.0:1789760774.375710:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760774.375723:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760774.375773:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27516928, count: 4096 00000080:00200000:2.0:1789760774.375782:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27516928, bytes: 4096 00000080:00200000:2.0:1789760774.375794:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760774.375804:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760774.375806:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27516928, 27521024) 0 0 00000080:00200000:2.0:1789760774.375811:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760774.375831:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6718, 6718] 00020000:00200000:2.0:1789760774.375841:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27516928 chunk: [27516928, 27521024) 27521024, 4096 00020000:00200000:2.0:1789760774.375846:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760774.375867:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6718, 6718] 00000008:00010000:2.0:1789760774.375886:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760774.375891:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760774.375934:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27516928, 27521024) append 0 append lockpos 0 00000080:00200000:2.0:1789760774.375950:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27516928=0x1a3e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760774.376016:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760774.376019:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27521024 00000020:00010000:2.0:1789760774.376028:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760774.376032:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760774.376035:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760774.376040:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760774.376043:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760774.376051:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760774.376058:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760774.376076:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760774.376194:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 27516928<>27521024 00000100:00100000:3.0:1789760774.376485:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512722816:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760774.433970:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974037 gen 3 00000008:00010000:2.0:1789760774.434023:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:2.0:1789760774.434031:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760774.434042:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760774.434069:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:2.0:1789760774.434190:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512722816:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760774.434389:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6718, 6718] 00000080:00200000:0.0:1789760774.434405:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.434411:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.434425:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27521024 i_blocks 55303 00000020:00200000:0.0:1789760774.434428:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.434438:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.434473:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27521024, count: 4096 00000080:00200000:0.0:1789760774.434481:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27521024, bytes: 4096 00000080:00200000:0.0:1789760774.434493:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.434501:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.434503:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27521024, 27525120) 0 0 00000080:00200000:0.0:1789760774.434508:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.434523:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6719, 6719] 00020000:00200000:0.0:1789760774.434531:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27521024 chunk: [27521024, 27525120) 27525120, 4096 00020000:00200000:0.0:1789760774.434536:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.434555:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6719, 6719] 00000008:00010000:0.0:1789760774.434574:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.434580:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760774.434592:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27521024, 27525120) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.434606:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27521024=0x1a3f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.434667:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.434670:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27525120 00000020:00010000:0.0:1789760774.434678:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.434681:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.434685:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.434688:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.434691:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.434699:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.434705:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.434721:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc300 00000008:00100000:0.0:1789760774.435518:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149500 - object 0x240000400:5 offset 27521024<>27525120 00000100:00100000:0.0:1789760774.435569:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760774.435592:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512722944:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760774.468390:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974038 gen 3 00000008:00010000:0.0:1789760774.468440:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:0.0:1789760774.468448:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.468459:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.468489:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:0.0:1789760774.468519:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512722944:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760774.469258:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6719, 6719] 00000080:00200000:2.0:1789760774.469277:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760774.469283:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760774.469298:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27525120 i_blocks 55303 00000020:00200000:2.0:1789760774.469303:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760774.469314:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760774.469349:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27525120, count: 4096 00000080:00200000:2.0:1789760774.469356:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27525120, bytes: 4096 00000080:00200000:2.0:1789760774.469370:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760774.469379:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760774.469381:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27525120, 27529216) 0 0 00000080:00200000:2.0:1789760774.469385:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760774.469402:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6720, 6720] 00020000:00200000:2.0:1789760774.469410:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27525120 chunk: [27525120, 27529216) 27529216, 4096 00020000:00200000:2.0:1789760774.469414:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760774.469435:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6720, 6720] 00000008:00010000:2.0:1789760774.469453:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760774.469460:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760774.469471:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27525120, 27529216) append 0 append lockpos 0 00000080:00200000:2.0:1789760774.469485:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27525120=0x1a40000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760774.469547:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760774.469550:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27529216 00000020:00010000:2.0:1789760774.469559:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760774.469563:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760774.469566:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760774.469571:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760774.469574:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760774.469583:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760774.469591:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760774.469612:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760774.469724:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 27525120<>27529216 00000100:00100000:2.0:1789760774.469779:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760774.469812:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512723072:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760774.512804:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974039 gen 3 00000008:00010000:2.0:1789760774.512855:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:2.0:1789760774.512865:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760774.512877:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760774.512981:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:2.0:1789760774.513011:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512723072:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760774.515585:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6720, 6720] 00000080:00200000:0.0:1789760774.515611:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.515618:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.515636:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27529216 i_blocks 55303 00000020:00200000:0.0:1789760774.515641:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.515658:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.515700:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27529216, count: 4096 00000080:00200000:0.0:1789760774.515709:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27529216, bytes: 4096 00000080:00200000:0.0:1789760774.515723:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.515732:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.515735:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27529216, 27533312) 0 0 00000080:00200000:0.0:1789760774.515739:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.515755:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6721, 6721] 00020000:00200000:0.0:1789760774.515764:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27529216 chunk: [27529216, 27533312) 27533312, 4096 00020000:00200000:0.0:1789760774.515768:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.515787:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6721, 6721] 00000008:00010000:0.0:1789760774.515804:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.515809:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760774.515820:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27529216, 27533312) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.515832:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27529216=0x1a41000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.515894:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.515897:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27533312 00000020:00010000:0.0:1789760774.515932:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.515937:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.515940:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.515943:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.515946:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.515954:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.515960:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.515989:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc300 00000008:00100000:0.0:1789760774.516103:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149500 - object 0x240000400:5 offset 27529216<>27533312 00000100:00100000:1.0:1789760774.516349:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512723200:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760774.548658:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974040 gen 3 00000008:00010000:1.0:1789760774.548713:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760774.548722:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.548734:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.548755:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760774.549023:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512723200:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760774.549190:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6721, 6721] 00000080:00200000:0.0:1789760774.549206:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.549212:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.549229:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27533312 i_blocks 55303 00000020:00200000:0.0:1789760774.549237:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.549248:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.549285:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27533312, count: 4096 00000080:00200000:0.0:1789760774.549293:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27533312, bytes: 4096 00000080:00200000:0.0:1789760774.549304:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.549313:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.549315:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27533312, 27537408) 0 0 00000080:00200000:0.0:1789760774.549320:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.549336:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6722, 6722] 00020000:00200000:0.0:1789760774.549344:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27533312 chunk: [27533312, 27537408) 27537408, 4096 00020000:00200000:0.0:1789760774.549349:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.549368:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6722, 6722] 00000008:00010000:0.0:1789760774.549385:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.549391:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760774.549403:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27533312, 27537408) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.549417:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27533312=0x1a42000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.549474:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.549477:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27537408 00000020:00010000:0.0:1789760774.549485:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.549489:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.549492:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.549496:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.549499:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.549506:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.549513:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.549530:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc240 00000008:00100000:0.0:1789760774.549832:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514bb80 - object 0x240000400:5 offset 27533312<>27537408 00000100:00100000:1.0:1789760774.549921:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512723328:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760774.583146:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974041 gen 3 00000008:00010000:1.0:1789760774.583207:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760774.583215:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.583225:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.583244:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760774.583267:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512723328:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760774.583359:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6722, 6722] 00000080:00200000:0.0:1789760774.583373:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.583378:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.583392:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27537408 i_blocks 55303 00000020:00200000:0.0:1789760774.583395:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.583403:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.583434:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27537408, count: 4096 00000080:00200000:0.0:1789760774.583439:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27537408, bytes: 4096 00000080:00200000:0.0:1789760774.583447:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.583455:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.583457:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27537408, 27541504) 0 0 00000080:00200000:0.0:1789760774.583461:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.583475:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6723, 6723] 00020000:00200000:0.0:1789760774.583483:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27537408 chunk: [27537408, 27541504) 27541504, 4096 00020000:00200000:0.0:1789760774.583486:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.583500:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6723, 6723] 00000008:00010000:0.0:1789760774.583514:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.583519:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760774.583529:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27537408, 27541504) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.583540:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27537408=0x1a43000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.583595:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.583597:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27541504 00000020:00010000:0.0:1789760774.583603:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.583607:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.583610:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.583614:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.583617:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.583624:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.583631:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.583647:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc480 00000008:00100000:0.0:1789760774.584124:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b100 - object 0x240000400:5 offset 27537408<>27541504 00000100:00100000:0.0:1789760774.584186:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512723456:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760774.629554:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974042 gen 3 00000008:00010000:0.0:1789760774.629614:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:0.0:1789760774.629627:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.629639:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.629712:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:0.0:1789760774.629741:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512723456:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760774.629978:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6723, 6723] 00000080:00200000:1.0:1789760774.629994:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.630000:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.630014:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27541504 i_blocks 55303 00000020:00200000:1.0:1789760774.630018:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.630028:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.630060:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27541504, count: 4096 00000080:00200000:1.0:1789760774.630066:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27541504, bytes: 4096 00000080:00200000:1.0:1789760774.630077:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.630096:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.630098:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27541504, 27545600) 0 0 00000080:00200000:1.0:1789760774.630102:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.630119:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6724, 6724] 00020000:00200000:1.0:1789760774.630127:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27541504 chunk: [27541504, 27545600) 27545600, 4096 00020000:00200000:1.0:1789760774.630131:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.630148:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6724, 6724] 00000008:00010000:1.0:1789760774.630163:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.630168:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760774.630189:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27541504, 27545600) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.630203:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27541504=0x1a44000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.630255:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.630258:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27545600 00000020:00010000:1.0:1789760774.630264:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.630268:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.630271:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.630274:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.630278:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.630284:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.630290:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.630307:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760774.630696:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 27541504<>27545600 00000100:00100000:0.0:1789760774.630771:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512723584:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760774.676516:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974043 gen 3 00000008:00010000:0.0:1789760774.676636:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760774.676646:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.676658:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.676741:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760774.676774:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512723584:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760774.677243:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6724, 6724] 00000080:00200000:1.0:1789760774.677312:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.677319:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.677334:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27545600 i_blocks 55303 00000020:00200000:1.0:1789760774.677338:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.677348:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.677385:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27545600, count: 4096 00000080:00200000:1.0:1789760774.677392:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27545600, bytes: 4096 00000080:00200000:1.0:1789760774.677405:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.677414:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.677416:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27545600, 27549696) 0 0 00000080:00200000:1.0:1789760774.677420:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.677438:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6725, 6725] 00020000:00200000:1.0:1789760774.677446:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27545600 chunk: [27545600, 27549696) 27549696, 4096 00020000:00200000:1.0:1789760774.677451:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.677708:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6725, 6725] 00000008:00010000:1.0:1789760774.677728:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.677735:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760774.677748:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27545600, 27549696) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.677763:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27545600=0x1a45000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.677837:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.677840:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27549696 00000020:00010000:1.0:1789760774.678133:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.678138:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.678141:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.678145:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.678149:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.678157:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.678166:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.678203:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760774.678392:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 27545600<>27549696 00000100:00100000:0.0:1789760774.678490:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512723712:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760774.722068:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974044 gen 3 00000008:00010000:0.0:1789760774.722123:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760774.722132:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.722145:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.722437:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000080:00200000:1.0:1789760774.722772:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6725, 6725] 00000080:00200000:1.0:1789760774.722796:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.722802:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.722820:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27549696 i_blocks 55303 00000020:00200000:1.0:1789760774.722824:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.722834:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.723013:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27549696, count: 4096 00000080:00200000:1.0:1789760774.723024:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27549696, bytes: 4096 00000080:00200000:1.0:1789760774.723037:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.723047:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.723049:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27549696, 27553792) 0 0 00000080:00200000:1.0:1789760774.723053:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.723073:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6726, 6726] 00020000:00200000:1.0:1789760774.723081:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27549696 chunk: [27549696, 27553792) 27553792, 4096 00020000:00200000:1.0:1789760774.723086:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.723107:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6726, 6726] 00000008:00010000:1.0:1789760774.723127:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.723133:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760774.723146:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27549696, 27553792) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.723160:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27549696=0x1a46000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.723321:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.723325:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27553792 00000020:00010000:1.0:1789760774.723333:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.723337:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.723340:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.723344:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.723348:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.723356:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000100:00100000:0.0:1789760774.723359:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512723712:192.168.203.156@tcp:4:dd.0 00000020:00200000:1.0:1789760774.723362:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.723416:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:0.0:1789760774.723539:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b100 - object 0x240000400:5 offset 27549696<>27553792 00000100:00100000:0.0:1789760774.723588:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760774.723604:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512723840:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760774.769048:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974045 gen 3 00000008:00010000:0.0:1789760774.769099:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760774.769108:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760774.769120:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760774.769139:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760774.769165:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512723840:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760774.769437:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6726, 6726] 00000080:00200000:1.0:1789760774.769453:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.769459:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.769475:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27553792 i_blocks 55303 00000020:00200000:1.0:1789760774.769479:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.769489:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.769524:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27553792, count: 4096 00000080:00200000:1.0:1789760774.769531:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27553792, bytes: 4096 00000080:00200000:1.0:1789760774.769541:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.769551:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.769553:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27553792, 27557888) 0 0 00000080:00200000:1.0:1789760774.769557:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.769574:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6727, 6727] 00020000:00200000:1.0:1789760774.769583:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27553792 chunk: [27553792, 27557888) 27557888, 4096 00020000:00200000:1.0:1789760774.769588:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.769608:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6727, 6727] 00000008:00010000:1.0:1789760774.769627:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.769633:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760774.769646:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27553792, 27557888) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.769658:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27553792=0x1a47000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.770087:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.770091:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27557888 00000020:00010000:1.0:1789760774.770100:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.770104:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.770107:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.770111:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.770115:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.770123:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.770131:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.770150:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760774.770347:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95c00 - object 0x240000400:5 offset 27553792<>27557888 00000100:00100000:1.0:1789760774.770399:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760774.770421:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512723968:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760774.809786:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974046 gen 3 00000008:00010000:1.0:1789760774.809839:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760774.809847:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.809858:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.809883:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760774.809952:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512723968:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760774.810109:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6727, 6727] 00000080:00200000:2.0:1789760774.810127:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760774.810133:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760774.810149:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27557888 i_blocks 55303 00000020:00200000:2.0:1789760774.810152:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760774.810162:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760774.810207:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27557888, count: 4096 00000080:00200000:2.0:1789760774.810215:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27557888, bytes: 4096 00000080:00200000:2.0:1789760774.810227:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760774.810235:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760774.810238:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27557888, 27561984) 0 0 00000080:00200000:2.0:1789760774.810242:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760774.810258:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6728, 6728] 00020000:00200000:2.0:1789760774.810266:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27557888 chunk: [27557888, 27561984) 27561984, 4096 00020000:00200000:2.0:1789760774.810270:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760774.810290:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6728, 6728] 00000008:00010000:2.0:1789760774.810306:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760774.810312:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760774.810323:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27557888, 27561984) append 0 append lockpos 0 00000080:00200000:2.0:1789760774.810336:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27557888=0x1a48000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760774.810392:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760774.810395:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27561984 00000020:00010000:2.0:1789760774.810416:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760774.810420:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760774.810423:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760774.810427:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760774.810430:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760774.810439:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760774.810447:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760774.810465:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760774.810570:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 27557888<>27561984 00000100:00100000:3.0:1789760774.810743:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760774.810764:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512724096:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760774.860011:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:2.0:1789760774.860027:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760774.860038:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760774.860064:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:2.0:1789760774.860102:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512724096:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760774.860339:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6728, 6728] 00000080:00200000:1.0:1789760774.860355:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760774.860362:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760774.860376:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27561984 i_blocks 55303 00000020:00200000:1.0:1789760774.860380:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760774.860390:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760774.860423:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27561984, count: 4096 00000080:00200000:1.0:1789760774.860431:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27561984, bytes: 4096 00000080:00200000:1.0:1789760774.860441:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760774.860450:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760774.860452:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27561984, 27566080) 0 0 00000080:00200000:1.0:1789760774.860457:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760774.860670:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6729, 6729] 00020000:00200000:1.0:1789760774.860680:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27561984 chunk: [27561984, 27566080) 27566080, 4096 00020000:00200000:1.0:1789760774.860684:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760774.860704:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6729, 6729] 00000008:00010000:1.0:1789760774.860722:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760774.860727:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760774.860739:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27561984, 27566080) append 0 append lockpos 0 00000080:00200000:1.0:1789760774.860752:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27561984=0x1a49000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760774.860811:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760774.860814:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27566080 00000020:00010000:1.0:1789760774.860945:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760774.860950:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.860953:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760774.860957:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760774.860960:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760774.860967:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760774.860976:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760774.860998:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760774.861183:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95c00 - object 0x240000400:5 offset 27561984<>27566080 00000100:00100000:1.0:1789760774.861243:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512724224:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760774.933702:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974048 gen 3 00000008:00010000:1.0:1789760774.933767:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760774.933777:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.933787:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.933812:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760774.933840:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512724224:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760774.933955:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6729, 6729] 00000080:00200000:0.0:1789760774.933972:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.933978:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.933995:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27566080 i_blocks 55303 00000020:00200000:0.0:1789760774.934000:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.934011:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.934047:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27566080, count: 4096 00000080:00200000:0.0:1789760774.934055:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27566080, bytes: 4096 00000080:00200000:0.0:1789760774.934066:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.934075:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.934078:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27566080, 27570176) 0 0 00000080:00200000:0.0:1789760774.934082:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.934101:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6730, 6730] 00020000:00200000:0.0:1789760774.934110:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27566080 chunk: [27566080, 27570176) 27570176, 4096 00020000:00200000:0.0:1789760774.934115:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.934136:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6730, 6730] 00000008:00010000:0.0:1789760774.934155:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.934161:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760774.934183:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27566080, 27570176) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.934197:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27566080=0x1a4a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.934258:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.934261:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27570176 00000020:00010000:0.0:1789760774.934270:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.934274:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.934278:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.934282:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.934285:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.934293:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.934300:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.934317:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc480 00000008:00100000:0.0:1789760774.934425:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b100 - object 0x240000400:5 offset 27566080<>27570176 00000100:00100000:1.0:1789760774.934596:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760774.934615:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512724352:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760774.941350:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512724608:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1789760774.941493:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:0.0:1789760774.941500:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e6cf7267-9b19-462a-a263-14115df15dab:2359:1876699512724480:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1789760774.941721:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2359:1876699512724736:192.168.203.156@tcp:400:kworker.0 00010000:00010000:2.0:1789760774.941807:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760774.953169:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e6cf7267-9b19-462a-a263-14115df15dab:2359:1876699512724480:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1789760774.953204:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2359:1876699512724736:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1789760774.953215:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512724608:192.168.203.156@tcp:400:kworker.0 00000008:00010000:1.0:1789760774.993950:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760774.993967:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760774.993979:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760774.994001:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760774.994031:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512724352:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760774.994288:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6730, 6730] 00000080:00200000:0.0:1789760774.994303:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760774.994309:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760774.994325:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27570176 i_blocks 55303 00000020:00200000:0.0:1789760774.994329:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760774.994339:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760774.994376:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27570176, count: 4096 00000080:00200000:0.0:1789760774.994384:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27570176, bytes: 4096 00000080:00200000:0.0:1789760774.994394:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760774.994403:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760774.994406:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27570176, 27574272) 0 0 00000080:00200000:0.0:1789760774.994410:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760774.994426:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6731, 6731] 00020000:00200000:0.0:1789760774.994435:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27570176 chunk: [27570176, 27574272) 27574272, 4096 00020000:00200000:0.0:1789760774.994439:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760774.994460:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6731, 6731] 00000008:00010000:0.0:1789760774.994479:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760774.994485:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760774.994497:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27570176, 27574272) append 0 append lockpos 0 00000080:00200000:0.0:1789760774.994511:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27570176=0x1a4b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760774.994572:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760774.994575:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27574272 00000020:00010000:0.0:1789760774.994583:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760774.994587:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.994590:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760774.994593:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760774.994596:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760774.994604:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760774.994610:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760774.994627:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc840 00000008:00100000:0.0:1789760774.994737:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149500 - object 0x240000400:5 offset 27570176<>27574272 00000100:00100000:1.0:1789760774.995016:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512724864:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760775.042628:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974050 gen 3 00000008:00010000:1.0:1789760775.042689:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760775.042696:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760775.042705:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760775.042722:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760775.042745:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512724864:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760775.043112:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6731, 6731] 00000080:00200000:0.0:1789760775.043133:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760775.043139:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760775.043156:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27574272 i_blocks 55303 00000020:00200000:0.0:1789760775.043160:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760775.043228:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760775.043263:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27574272, count: 4096 00000080:00200000:0.0:1789760775.043271:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27574272, bytes: 4096 00000080:00200000:0.0:1789760775.043284:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760775.043293:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760775.043295:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27574272, 27578368) 0 0 00000080:00200000:0.0:1789760775.043299:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760775.043318:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6732, 6732] 00020000:00200000:0.0:1789760775.043327:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27574272 chunk: [27574272, 27578368) 27578368, 4096 00020000:00200000:0.0:1789760775.043331:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760775.043351:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6732, 6732] 00000008:00010000:0.0:1789760775.043369:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760775.043374:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760775.043388:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27574272, 27578368) append 0 append lockpos 0 00000080:00200000:0.0:1789760775.043404:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27574272=0x1a4c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760775.043471:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760775.043474:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27578368 00000020:00010000:0.0:1789760775.043483:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760775.043487:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.043490:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.043493:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760775.043495:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760775.043503:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760775.043510:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760775.043531:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc840 00000008:00100000:0.0:1789760775.043772:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149f80 - object 0x240000400:5 offset 27574272<>27578368 00000100:00100000:1.0:1789760775.043968:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512724992:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760775.085514:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974051 gen 3 00000008:00010000:1.0:1789760775.085577:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760775.085585:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760775.085596:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760775.085617:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760775.085645:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512724992:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760775.085978:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6732, 6732] 00000080:00200000:0.0:1789760775.085996:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760775.086001:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760775.086017:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27578368 i_blocks 55303 00000020:00200000:0.0:1789760775.086021:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760775.086030:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760775.086064:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27578368, count: 4096 00000080:00200000:0.0:1789760775.086071:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27578368, bytes: 4096 00000080:00200000:0.0:1789760775.086083:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760775.086092:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760775.086094:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27578368, 27582464) 0 0 00000080:00200000:0.0:1789760775.086098:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760775.086116:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6733, 6733] 00020000:00200000:0.0:1789760775.086127:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27578368 chunk: [27578368, 27582464) 27582464, 4096 00020000:00200000:0.0:1789760775.086131:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760775.086150:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6733, 6733] 00000008:00010000:0.0:1789760775.086168:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760775.086533:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760775.086546:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27578368, 27582464) append 0 append lockpos 0 00000080:00200000:0.0:1789760775.086560:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27578368=0x1a4d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760775.086729:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760775.086732:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27582464 00000020:00010000:0.0:1789760775.086742:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760775.086745:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.086748:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.086751:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760775.086754:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760775.086763:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760775.086770:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760775.087640:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc480 00000008:00100000:0.0:1789760775.087770:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514bb80 - object 0x240000400:5 offset 27578368<>27582464 00000100:00100000:1.0:1789760775.088221:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512725120:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760775.140201:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974052 gen 3 00000008:00010000:1.0:1789760775.140255:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760775.140263:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760775.140275:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760775.140356:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760775.140390:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512725120:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760775.140487:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6733, 6733] 00000080:00200000:0.0:1789760775.140505:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760775.140511:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760775.140530:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27582464 i_blocks 55303 00000020:00200000:0.0:1789760775.140533:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760775.140544:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760775.140584:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27582464, count: 4096 00000080:00200000:0.0:1789760775.140592:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27582464, bytes: 4096 00000080:00200000:0.0:1789760775.140603:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760775.140612:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760775.140615:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27582464, 27586560) 0 0 00000080:00200000:0.0:1789760775.140619:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760775.140638:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6734, 6734] 00020000:00200000:0.0:1789760775.140647:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27582464 chunk: [27582464, 27586560) 27586560, 4096 00020000:00200000:0.0:1789760775.140651:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760775.140673:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6734, 6734] 00000008:00010000:0.0:1789760775.140694:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760775.140701:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760775.140712:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27582464, 27586560) append 0 append lockpos 0 00000080:00200000:0.0:1789760775.140726:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27582464=0x1a4e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760775.140788:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760775.140792:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27586560 00000020:00010000:0.0:1789760775.140801:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760775.140807:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.140810:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.140813:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760775.140817:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760775.140825:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760775.140833:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760775.141147:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc240 00000008:00100000:0.0:1789760775.141266:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149c00 - object 0x240000400:5 offset 27582464<>27586560 00000100:00100000:1.0:1789760775.141330:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512725248:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760775.206653:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974053 gen 3 00000008:00010000:1.0:1789760775.206705:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760775.206712:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760775.206722:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760775.206836:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000080:00200000:2.0:1789760775.206837:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6734, 6734] 00000080:00200000:2.0:1789760775.206855:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.206861:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000100:00100000:1.0:1789760775.206868:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512725248:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760775.206876:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27586560 i_blocks 55303 00000020:00200000:2.0:1789760775.206879:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.206888:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.206948:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27586560, count: 4096 00000080:00200000:2.0:1789760775.206957:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27586560, bytes: 4096 00000080:00200000:2.0:1789760775.206968:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.206976:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.206978:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27586560, 27590656) 0 0 00000080:00200000:2.0:1789760775.206983:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.207000:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6735, 6735] 00020000:00200000:2.0:1789760775.207009:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27586560 chunk: [27586560, 27590656) 27590656, 4096 00020000:00200000:2.0:1789760775.207014:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.207034:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6735, 6735] 00000008:00010000:2.0:1789760775.207052:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.207057:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760775.207066:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27586560, 27590656) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.207081:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27586560=0x1a4f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.207141:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.207143:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27590656 00000020:00010000:2.0:1789760775.207152:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.207156:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.207159:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.207162:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.207165:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.207186:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.207194:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.207211:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760775.207330:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 27586560<>27590656 00000100:00100000:3.0:1789760775.207451:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512725376:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760775.255960:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760775.255977:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.255987:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.256009:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760775.256038:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512725376:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760775.256646:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6735, 6735] 00000080:00200000:2.0:1789760775.256665:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.256671:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.256687:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27590656 i_blocks 55303 00000020:00200000:2.0:1789760775.256691:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.256701:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.256803:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27590656, count: 4096 00000080:00200000:2.0:1789760775.256812:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27590656, bytes: 4096 00000080:00200000:2.0:1789760775.256826:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.256836:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.256838:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27590656, 27594752) 0 0 00000080:00200000:2.0:1789760775.256843:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.256859:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6736, 6736] 00020000:00200000:2.0:1789760775.256868:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27590656 chunk: [27590656, 27594752) 27594752, 4096 00020000:00200000:2.0:1789760775.256872:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.256891:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6736, 6736] 00000008:00010000:2.0:1789760775.256942:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.256949:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760775.256961:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27590656, 27594752) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.256976:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27590656=0x1a50000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.257038:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.257041:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27594752 00000020:00010000:2.0:1789760775.257050:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.257053:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.257056:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.257060:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.257063:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.257071:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.257078:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.257099:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760775.257955:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 27590656<>27594752 00000100:00100000:3.0:1789760775.258456:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512725504:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760775.301238:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974055 gen 3 00000008:00010000:3.0:1789760775.301308:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760775.301317:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.301327:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.302401:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000080:00200000:2.0:1789760775.302423:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6736, 6736] 00000080:00200000:2.0:1789760775.302442:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1789760775.302442:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512725504:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760775.302448:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.302463:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27594752 i_blocks 55303 00000020:00200000:2.0:1789760775.302468:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.302479:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.302514:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27594752, count: 4096 00000080:00200000:2.0:1789760775.302521:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27594752, bytes: 4096 00000080:00200000:2.0:1789760775.302534:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.302542:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.302544:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27594752, 27598848) 0 0 00000080:00200000:2.0:1789760775.302549:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.302565:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6737, 6737] 00020000:00200000:2.0:1789760775.302574:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27594752 chunk: [27594752, 27598848) 27598848, 4096 00020000:00200000:2.0:1789760775.302579:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.302598:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6737, 6737] 00000008:00010000:2.0:1789760775.302614:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.302620:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760775.302647:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27594752, 27598848) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.302663:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27594752=0x1a51000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.302733:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.302736:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27598848 00000020:00010000:2.0:1789760775.302744:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.302748:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.302751:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.302754:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.302758:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.302764:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.302772:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.302788:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760775.302924:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 27594752<>27598848 00000100:00100000:3.0:1789760775.303075:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512725632:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760775.340793:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760775.340811:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.340823:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.340842:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760775.340879:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512725632:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760775.341405:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6737, 6737] 00000080:00200000:2.0:1789760775.341426:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.341432:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.341462:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27598848 i_blocks 55303 00000020:00200000:2.0:1789760775.341467:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.341482:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.341522:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27598848, count: 4096 00000080:00200000:2.0:1789760775.341530:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27598848, bytes: 4096 00000080:00200000:2.0:1789760775.341541:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.341551:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.341553:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27598848, 27602944) 0 0 00000080:00200000:2.0:1789760775.341558:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.341580:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6738, 6738] 00020000:00200000:2.0:1789760775.341588:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27598848 chunk: [27598848, 27602944) 27602944, 4096 00020000:00200000:2.0:1789760775.341593:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.341615:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6738, 6738] 00000008:00010000:2.0:1789760775.341633:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.341638:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760775.343487:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27598848, 27602944) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.343515:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27598848=0x1a52000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.343608:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.343612:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27602944 00000020:00010000:2.0:1789760775.343622:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.343625:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.343630:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.343633:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.343636:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.343647:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.343656:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.343688:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760775.343817:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 27598848<>27602944 00000100:00100000:3.0:1789760775.345007:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512725760:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760775.400451:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974057 gen 3 00000008:00010000:3.0:1789760775.400513:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760775.400521:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.400538:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.400565:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760775.400597:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512725760:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760775.400945:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6738, 6738] 00000080:00200000:2.0:1789760775.400963:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.400969:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.400986:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27602944 i_blocks 55303 00000020:00200000:2.0:1789760775.400989:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.401000:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.401035:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27602944, count: 4096 00000080:00200000:2.0:1789760775.401043:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27602944, bytes: 4096 00000080:00200000:2.0:1789760775.401053:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.401062:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.401065:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27602944, 27607040) 0 0 00000080:00200000:2.0:1789760775.401069:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.401086:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6739, 6739] 00020000:00200000:2.0:1789760775.401105:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27602944 chunk: [27602944, 27607040) 27607040, 4096 00020000:00200000:2.0:1789760775.401110:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.401131:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6739, 6739] 00000008:00010000:2.0:1789760775.401149:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.401154:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760775.401166:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27602944, 27607040) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.401195:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27602944=0x1a53000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.401256:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.401259:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27607040 00000020:00010000:2.0:1789760775.401267:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.401271:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.401273:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.401276:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.401279:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.401287:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.401294:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.401311:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760775.401413:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 27602944<>27607040 00000100:00100000:3.0:1789760775.402284:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760775.402320:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512725888:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760775.437227:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974058 gen 3 00000008:00010000:3.0:1789760775.437277:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760775.437285:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.437296:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.437317:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760775.437349:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512725888:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760775.437450:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6739, 6739] 00000080:00200000:2.0:1789760775.437466:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.437472:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.437487:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27607040 i_blocks 55303 00000020:00200000:2.0:1789760775.437491:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.437501:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.437546:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27607040, count: 4096 00000080:00200000:2.0:1789760775.437552:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27607040, bytes: 4096 00000080:00200000:2.0:1789760775.437568:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.437577:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.437579:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27607040, 27611136) 0 0 00000080:00200000:2.0:1789760775.437584:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.437600:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6740, 6740] 00020000:00200000:2.0:1789760775.437608:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27607040 chunk: [27607040, 27611136) 27611136, 4096 00020000:00200000:2.0:1789760775.437613:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.437632:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6740, 6740] 00000008:00010000:2.0:1789760775.437650:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.437656:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760775.437668:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27607040, 27611136) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.437683:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27607040=0x1a54000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.437743:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.437746:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27611136 00000020:00010000:2.0:1789760775.437754:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.437757:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.437760:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.437764:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.437766:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.437774:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.437780:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.437796:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760775.438165:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 27607040<>27611136 00000100:00100000:2.0:1789760775.438221:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760775.438241:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512726016:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760775.497379:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974059 gen 3 00000008:00010000:3.0:1789760775.497433:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760775.497443:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760775.497456:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760775.497480:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760775.497511:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512726016:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760775.497852:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6740, 6740] 00000080:00200000:1.0:1789760775.497872:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760775.497878:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760775.497937:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27611136 i_blocks 55303 00000020:00200000:1.0:1789760775.497943:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760775.497967:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760775.497996:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27611136, count: 4096 00000080:00200000:1.0:1789760775.498003:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27611136, bytes: 4096 00000080:00200000:1.0:1789760775.498014:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760775.498022:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760775.498024:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27611136, 27615232) 0 0 00000080:00200000:1.0:1789760775.498027:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760775.498041:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6741, 6741] 00020000:00200000:1.0:1789760775.498047:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27611136 chunk: [27611136, 27615232) 27615232, 4096 00020000:00200000:1.0:1789760775.498051:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760775.498067:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6741, 6741] 00000008:00010000:1.0:1789760775.498082:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760775.498087:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760775.498098:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27611136, 27615232) append 0 append lockpos 0 00000080:00200000:1.0:1789760775.498110:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27611136=0x1a55000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760775.498165:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760775.498177:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27615232 00000020:00010000:1.0:1789760775.498185:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760775.498188:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.498191:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.498194:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760775.498197:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760775.498203:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760775.498210:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760775.498229:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760775.498316:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95c00 - object 0x240000400:5 offset 27611136<>27615232 00000100:00100000:1.0:1789760775.498366:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760775.498390:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512726144:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760775.547856:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974060 gen 3 00000008:00010000:1.0:1789760775.548062:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760775.548070:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760775.548080:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760775.548267:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760775.548295:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512726144:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760775.548457:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6741, 6741] 00000080:00200000:0.0:1789760775.548472:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760775.548478:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760775.548494:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27615232 i_blocks 55303 00000020:00200000:0.0:1789760775.548497:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760775.548507:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760775.548645:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27615232, count: 4096 00000080:00200000:0.0:1789760775.548653:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27615232, bytes: 4096 00000080:00200000:0.0:1789760775.548664:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760775.548673:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760775.548675:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27615232, 27619328) 0 0 00000080:00200000:0.0:1789760775.548680:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760775.548697:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6742, 6742] 00020000:00200000:0.0:1789760775.548704:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27615232 chunk: [27615232, 27619328) 27619328, 4096 00020000:00200000:0.0:1789760775.548708:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760775.548725:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6742, 6742] 00000008:00010000:0.0:1789760775.548741:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760775.548747:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760775.548757:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27615232, 27619328) append 0 append lockpos 0 00000080:00200000:0.0:1789760775.548769:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27615232=0x1a56000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760775.548829:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760775.548832:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27619328 00000020:00010000:0.0:1789760775.548839:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760775.548843:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.548846:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.548849:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760775.548852:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760775.548859:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760775.548865:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760775.548882:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc300 00000008:00100000:0.0:1789760775.549018:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148000 - object 0x240000400:5 offset 27615232<>27619328 00000100:00100000:0.0:1789760775.549051:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760775.549067:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512726272:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760775.595316:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974061 gen 3 00000008:00010000:0.0:1789760775.595375:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:0.0:1789760775.595383:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760775.595394:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760775.595464:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:0.0:1789760775.595491:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512726272:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760775.595866:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6742, 6742] 00000080:00200000:1.0:1789760775.595882:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760775.595888:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760775.595936:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27619328 i_blocks 55303 00000020:00200000:1.0:1789760775.595940:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760775.595950:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760775.595983:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27619328, count: 4096 00000080:00200000:1.0:1789760775.595990:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27619328, bytes: 4096 00000080:00200000:1.0:1789760775.596002:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760775.596011:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760775.596013:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27619328, 27623424) 0 0 00000080:00200000:1.0:1789760775.596018:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760775.596036:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6743, 6743] 00020000:00200000:1.0:1789760775.596045:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27619328 chunk: [27619328, 27623424) 27623424, 4096 00020000:00200000:1.0:1789760775.596050:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760775.596071:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6743, 6743] 00000008:00010000:1.0:1789760775.596089:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760775.596095:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760775.596105:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27619328, 27623424) append 0 append lockpos 0 00000080:00200000:1.0:1789760775.596118:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27619328=0x1a57000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760775.596186:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760775.596189:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27623424 00000020:00010000:1.0:1789760775.596198:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760775.596201:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.596204:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.596207:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760775.596210:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760775.596216:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760775.596223:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760775.596242:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760775.596343:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95c00 - object 0x240000400:5 offset 27619328<>27623424 00000100:00100000:0.0:1789760775.596412:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512726400:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760775.650400:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974062 gen 3 00000008:00010000:0.0:1789760775.650448:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760775.650457:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760775.650467:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760775.650540:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760775.650578:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512726400:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760775.652355:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6743, 6743] 00000080:00200000:1.0:1789760775.652380:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760775.652386:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760775.652403:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27623424 i_blocks 55303 00000020:00200000:1.0:1789760775.652408:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760775.652422:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760775.652458:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27623424, count: 4096 00000080:00200000:1.0:1789760775.652466:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27623424, bytes: 4096 00000080:00200000:1.0:1789760775.652480:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760775.652489:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760775.652492:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27623424, 27627520) 0 0 00000080:00200000:1.0:1789760775.652496:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760775.652513:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6744, 6744] 00020000:00200000:1.0:1789760775.652522:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27623424 chunk: [27623424, 27627520) 27627520, 4096 00020000:00200000:1.0:1789760775.652645:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760775.652667:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6744, 6744] 00000008:00010000:1.0:1789760775.652686:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760775.652692:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760775.652704:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27623424, 27627520) append 0 append lockpos 0 00000080:00200000:1.0:1789760775.652718:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27623424=0x1a58000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760775.652885:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760775.652888:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27627520 00000020:00010000:1.0:1789760775.653089:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760775.653094:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.653097:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.653101:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760775.653104:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760775.653113:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760775.653121:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760775.653151:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760775.653340:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 27623424<>27627520 00000100:00100000:0.0:1789760775.653422:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512726528:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760775.713885:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974063 gen 3 00000008:00010000:0.0:1789760775.713975:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760775.713984:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760775.713995:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760775.714019:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760775.714050:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512726528:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760775.714249:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6744, 6744] 00000080:00200000:1.0:1789760775.714266:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760775.714281:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760775.714297:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27627520 i_blocks 55303 00000020:00200000:1.0:1789760775.714302:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760775.714312:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760775.714348:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27627520, count: 4096 00000080:00200000:1.0:1789760775.714357:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27627520, bytes: 4096 00000080:00200000:1.0:1789760775.714367:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760775.714375:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760775.714377:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27627520, 27631616) 0 0 00000080:00200000:1.0:1789760775.714381:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760775.714398:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6745, 6745] 00020000:00200000:1.0:1789760775.714406:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27627520 chunk: [27627520, 27631616) 27631616, 4096 00020000:00200000:1.0:1789760775.714411:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760775.714433:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6745, 6745] 00000008:00010000:1.0:1789760775.714449:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760775.714454:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760775.714465:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27627520, 27631616) append 0 append lockpos 0 00000080:00200000:1.0:1789760775.714478:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27627520=0x1a59000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760775.714529:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760775.714531:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27631616 00000020:00010000:1.0:1789760775.714539:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760775.714543:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.714546:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.714549:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760775.714552:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760775.714560:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760775.714566:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760775.714877:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:0.0:1789760775.715004:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148000 - object 0x240000400:5 offset 27627520<>27631616 00000100:00100000:0.0:1789760775.715043:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760775.715062:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512726656:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760775.777156:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974064 gen 3 00000008:00010000:0.0:1789760775.777784:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760775.777796:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760775.777809:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760775.777960:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760775.778006:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512726656:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760775.778347:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6745, 6745] 00000080:00200000:1.0:1789760775.778364:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760775.778370:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760775.778476:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27631616 i_blocks 55303 00000020:00200000:1.0:1789760775.778481:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760775.778493:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760775.778527:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27631616, count: 4096 00000080:00200000:1.0:1789760775.778535:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27631616, bytes: 4096 00000080:00200000:1.0:1789760775.778548:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760775.778612:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760775.778616:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27631616, 27635712) 0 0 00000080:00200000:1.0:1789760775.778621:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760775.778637:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6746, 6746] 00020000:00200000:1.0:1789760775.778645:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27631616 chunk: [27631616, 27635712) 27635712, 4096 00020000:00200000:1.0:1789760775.778706:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760775.778730:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6746, 6746] 00000008:00010000:1.0:1789760775.778748:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760775.778753:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760775.778766:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27631616, 27635712) append 0 append lockpos 0 00000080:00200000:1.0:1789760775.778780:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27631616=0x1a5a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760775.778841:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760775.778845:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27635712 00000020:00010000:1.0:1789760775.778853:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760775.778857:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.778860:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.778864:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760775.778867:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760775.778874:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760775.778880:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760775.778995:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760775.779177:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 27631616<>27635712 00000100:00100000:0.0:1789760775.779251:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512726784:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760775.830395:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974065 gen 3 00000008:00010000:0.0:1789760775.830457:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760775.830465:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760775.830477:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760775.830493:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760775.830522:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512726784:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760775.831053:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6746, 6746] 00000080:00200000:1.0:1789760775.831071:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760775.831077:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760775.831095:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27635712 i_blocks 55303 00000020:00200000:1.0:1789760775.831098:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760775.831108:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760775.831138:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27635712, count: 4096 00000080:00200000:1.0:1789760775.831146:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27635712, bytes: 4096 00000080:00200000:1.0:1789760775.831156:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760775.831346:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760775.831349:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27635712, 27639808) 0 0 00000080:00200000:1.0:1789760775.831355:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760775.831372:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6747, 6747] 00020000:00200000:1.0:1789760775.831381:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27635712 chunk: [27635712, 27639808) 27639808, 4096 00020000:00200000:1.0:1789760775.831385:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760775.831408:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6747, 6747] 00000008:00010000:1.0:1789760775.831428:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760775.831434:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760775.831446:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27635712, 27639808) append 0 append lockpos 0 00000080:00200000:1.0:1789760775.831461:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27635712=0x1a5b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760775.831515:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760775.831518:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27639808 00000020:00010000:1.0:1789760775.831525:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760775.831528:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.831530:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.831532:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760775.831534:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760775.831540:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760775.831545:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760775.831557:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760775.831662:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97100 - object 0x240000400:5 offset 27635712<>27639808 00000100:00100000:0.0:1789760775.831757:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512726912:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760775.867379:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760775.867397:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760775.867409:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760775.868186:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760775.868229:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512726912:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760775.868344:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6747, 6747] 00000080:00200000:1.0:1789760775.868362:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760775.868368:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760775.868383:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27639808 i_blocks 55303 00000020:00200000:1.0:1789760775.868388:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760775.868398:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760775.868432:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27639808, count: 4096 00000080:00200000:1.0:1789760775.868439:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27639808, bytes: 4096 00000080:00200000:1.0:1789760775.868451:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760775.868579:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760775.868582:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27639808, 27643904) 0 0 00000080:00200000:1.0:1789760775.868587:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760775.868729:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6748, 6748] 00020000:00200000:1.0:1789760775.868741:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27639808 chunk: [27639808, 27643904) 27643904, 4096 00020000:00200000:1.0:1789760775.868746:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760775.868768:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6748, 6748] 00000008:00010000:1.0:1789760775.868893:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760775.868920:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760775.868936:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27639808, 27643904) append 0 append lockpos 0 00000080:00200000:1.0:1789760775.868952:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27639808=0x1a5c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760775.869022:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760775.869026:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27643904 00000020:00010000:1.0:1789760775.869034:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760775.869038:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.869041:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760775.869045:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760775.869048:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760775.869057:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760775.869064:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760775.869369:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760775.869507:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 27639808<>27643904 00000100:00100000:1.0:1789760775.869627:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760775.869650:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512727040:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760775.916138:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974067 gen 3 00000008:00010000:1.0:1789760775.916246:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760775.916254:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760775.916264:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760775.916348:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760775.916378:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512727040:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760775.916718:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6748, 6748] 00000080:00200000:0.0:1789760775.916734:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760775.916740:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760775.916755:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27643904 i_blocks 55303 00000020:00200000:0.0:1789760775.916759:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760775.916769:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760775.917041:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27643904, count: 4096 00000080:00200000:0.0:1789760775.917051:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27643904, bytes: 4096 00000080:00200000:0.0:1789760775.917063:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760775.917072:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760775.917075:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27643904, 27648000) 0 0 00000080:00200000:0.0:1789760775.917080:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760775.917098:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6749, 6749] 00020000:00200000:0.0:1789760775.917106:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27643904 chunk: [27643904, 27648000) 27648000, 4096 00020000:00200000:0.0:1789760775.917110:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760775.917127:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6749, 6749] 00000008:00010000:0.0:1789760775.917144:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760775.917149:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760775.917161:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27643904, 27648000) append 0 append lockpos 0 00000080:00200000:0.0:1789760775.917184:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27643904=0x1a5d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760775.917243:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760775.917246:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27648000 00000020:00010000:0.0:1789760775.917254:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760775.917257:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.917260:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760775.917263:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760775.917266:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760775.917274:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760775.917280:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760775.917301:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc300 00000008:00100000:0.0:1789760775.917413:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148000 - object 0x240000400:5 offset 27643904<>27648000 00000100:00100000:1.0:1789760775.917498:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512727168:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760775.961433:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974068 gen 3 00000008:00010000:1.0:1789760775.961485:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760775.961494:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760775.961506:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760775.961568:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760775.961600:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512727168:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760775.961695:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6749, 6749] 00000080:00200000:2.0:1789760775.961712:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760775.961718:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760775.961733:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27648000 i_blocks 55303 00000020:00200000:2.0:1789760775.961736:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760775.961746:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760775.961780:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27648000, count: 4096 00000080:00200000:2.0:1789760775.961942:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27648000, bytes: 4096 00000080:00200000:2.0:1789760775.961955:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760775.961964:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760775.961966:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27648000, 27652096) 0 0 00000080:00200000:2.0:1789760775.961970:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760775.961988:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6750, 6750] 00020000:00200000:2.0:1789760775.961997:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27648000 chunk: [27648000, 27652096) 27652096, 4096 00020000:00200000:2.0:1789760775.962001:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760775.962022:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6750, 6750] 00000008:00010000:2.0:1789760775.962040:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760775.962046:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760775.962057:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27648000, 27652096) append 0 append lockpos 0 00000080:00200000:2.0:1789760775.962070:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27648000=0x1a5e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760775.962130:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760775.962133:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27652096 00000020:00010000:2.0:1789760775.962141:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760775.962145:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.962148:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760775.962152:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760775.962155:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760775.962163:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760775.962180:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760775.962198:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760775.962305:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 27648000<>27652096 00000100:00100000:3.0:1789760775.962381:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512727296:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760775.965460:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760776.001955:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974069 gen 3 00000008:00010000:3.0:1789760776.002007:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760776.002014:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.002025:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.002046:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000080:00200000:2.0:1789760776.002157:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6750, 6750] 00000080:00200000:2.0:1789760776.002184:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.002190:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.002207:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27652096 i_blocks 55303 00000100:00100000:3.0:1789760776.002290:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512727296:192.168.203.156@tcp:4:dd.0 00000020:00200000:2.0:1789760776.002378:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.002388:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.002423:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27652096, count: 4096 00000080:00200000:2.0:1789760776.002432:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27652096, bytes: 4096 00000080:00200000:2.0:1789760776.002442:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.002450:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.002452:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27652096, 27656192) 0 0 00000080:00200000:2.0:1789760776.002457:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.002474:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6751, 6751] 00020000:00200000:2.0:1789760776.002483:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27652096 chunk: [27652096, 27656192) 27656192, 4096 00020000:00200000:2.0:1789760776.002489:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.002510:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6751, 6751] 00000008:00010000:2.0:1789760776.002529:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.002535:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760776.002546:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27652096, 27656192) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.004015:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27652096=0x1a5f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.004709:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.004714:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27656192 00000020:00010000:2.0:1789760776.004726:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.004729:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.004733:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.004737:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.004740:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.004751:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.004759:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.004776:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760776.005342:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 27652096<>27656192 00000100:00100000:2.0:1789760776.005404:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512727424:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760776.049427:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760776.049437:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.049446:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.049512:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760776.049533:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512727424:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760776.049585:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6751, 6751] 00000080:00200000:0.0:1789760776.049596:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760776.049601:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760776.049612:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27656192 i_blocks 55303 00000020:00200000:0.0:1789760776.049615:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760776.049622:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760776.049645:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27656192, count: 4096 00000080:00200000:0.0:1789760776.049650:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27656192, bytes: 4096 00000080:00200000:0.0:1789760776.049658:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760776.049665:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760776.049667:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27656192, 27660288) 0 0 00000080:00200000:0.0:1789760776.049671:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760776.049682:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6752, 6752] 00020000:00200000:0.0:1789760776.049688:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27656192 chunk: [27656192, 27660288) 27660288, 4096 00020000:00200000:0.0:1789760776.049691:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760776.049704:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6752, 6752] 00000008:00010000:0.0:1789760776.049715:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760776.049719:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760776.049727:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27656192, 27660288) append 0 append lockpos 0 00000080:00200000:0.0:1789760776.049736:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27656192=0x1a60000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760776.049772:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760776.049774:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27660288 00000020:00010000:0.0:1789760776.049780:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760776.049783:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760776.049786:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760776.049789:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760776.049792:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760776.049798:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760776.049802:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760776.049814:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccb40 00000008:00100000:0.0:1789760776.050080:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149180 - object 0x240000400:5 offset 27656192<>27660288 00000100:00100000:1.0:1789760776.050127:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760776.050339:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512727552:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760776.092038:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974071 gen 3 00000008:00010000:1.0:1789760776.092106:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760776.092115:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760776.092126:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760776.092213:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760776.092254:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512727552:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760776.092394:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6752, 6752] 00000080:00200000:3.0:1789760776.092411:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760776.092417:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760776.092431:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27660288 i_blocks 55303 00000020:00200000:3.0:1789760776.092435:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760776.092444:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760776.092481:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27660288, count: 4096 00000080:00200000:3.0:1789760776.092488:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27660288, bytes: 4096 00000080:00200000:3.0:1789760776.092500:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760776.092509:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760776.092511:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27660288, 27664384) 0 0 00000080:00200000:3.0:1789760776.092515:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760776.092532:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6753, 6753] 00020000:00200000:3.0:1789760776.092542:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27660288 chunk: [27660288, 27664384) 27664384, 4096 00020000:00200000:3.0:1789760776.092546:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760776.092636:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6753, 6753] 00000008:00010000:3.0:1789760776.092659:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760776.092666:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760776.092679:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27660288, 27664384) append 0 append lockpos 0 00000080:00200000:3.0:1789760776.092694:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27660288=0x1a61000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760776.092753:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760776.092756:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27664384 00000020:00010000:3.0:1789760776.092765:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760776.092768:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.092771:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.092775:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760776.092778:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760776.092785:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760776.093090:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760776.093112:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760776.093305:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 27660288<>27664384 00000100:00100000:2.0:1789760776.093496:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512727680:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760776.147560:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974072 gen 3 00000008:00010000:3.0:1789760776.147615:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760776.147624:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760776.147637:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760776.148414:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000080:00200000:1.0:1789760776.148431:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6753, 6753] 00000080:00200000:1.0:1789760776.148452:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.148459:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.148476:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27664384 i_blocks 55303 00000020:00200000:1.0:1789760776.148480:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.148491:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000100:00100000:3.0:1789760776.148521:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512727680:192.168.203.156@tcp:4:dd.0 00000080:00200008:1.0:1789760776.148527:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27664384, count: 4096 00000080:00200000:1.0:1789760776.148537:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27664384, bytes: 4096 00000080:00200000:1.0:1789760776.148551:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.148560:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.148563:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27664384, 27668480) 0 0 00000080:00200000:1.0:1789760776.148568:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.148585:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6754, 6754] 00020000:00200000:1.0:1789760776.148595:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27664384 chunk: [27664384, 27668480) 27668480, 4096 00020000:00200000:1.0:1789760776.148600:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.148621:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6754, 6754] 00000008:00010000:1.0:1789760776.148639:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.148645:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760776.148657:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27664384, 27668480) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.148671:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27664384=0x1a62000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.148733:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.148736:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27668480 00000020:00010000:1.0:1789760776.148746:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.148750:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.148754:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.148757:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.148760:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.148781:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.148979:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.148999:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760776.149116:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 27664384<>27668480 00000100:00100000:0.0:1789760776.149239:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512727808:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760776.195179:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760776.195195:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.195206:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.195303:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760776.195343:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512727808:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760776.195558:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6754, 6754] 00000080:00200000:1.0:1789760776.195575:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.195582:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.195598:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27668480 i_blocks 55303 00000020:00200000:1.0:1789760776.195601:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.195611:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.195647:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27668480, count: 4096 00000080:00200000:1.0:1789760776.195654:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27668480, bytes: 4096 00000080:00200000:1.0:1789760776.195666:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.195675:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.195677:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27668480, 27672576) 0 0 00000080:00200000:1.0:1789760776.195681:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.195698:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6755, 6755] 00020000:00200000:1.0:1789760776.195706:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27668480 chunk: [27668480, 27672576) 27672576, 4096 00020000:00200000:1.0:1789760776.195710:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.195728:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6755, 6755] 00000008:00010000:1.0:1789760776.195747:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.195753:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760776.195763:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27668480, 27672576) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.195777:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27668480=0x1a63000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.196348:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.196353:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27672576 00000020:00010000:1.0:1789760776.196362:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.196366:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.196370:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.196374:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.196377:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.196387:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.196394:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.196414:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760776.196526:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97100 - object 0x240000400:5 offset 27668480<>27672576 00000100:00100000:0.0:1789760776.197363:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512727936:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760776.234196:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974074 gen 3 00000008:00010000:0.0:1789760776.234272:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760776.234281:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.234292:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.234319:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760776.234356:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512727936:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760776.234608:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6755, 6755] 00000080:00200000:1.0:1789760776.234626:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.234632:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.234648:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27672576 i_blocks 55303 00000020:00200000:1.0:1789760776.234653:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.234663:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.234699:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27672576, count: 4096 00000080:00200000:1.0:1789760776.234707:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27672576, bytes: 4096 00000080:00200000:1.0:1789760776.234716:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.234724:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.234726:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27672576, 27676672) 0 0 00000080:00200000:1.0:1789760776.234730:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.234924:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6756, 6756] 00020000:00200000:1.0:1789760776.234934:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27672576 chunk: [27672576, 27676672) 27676672, 4096 00020000:00200000:1.0:1789760776.234938:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.234959:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6756, 6756] 00000008:00010000:1.0:1789760776.234976:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.234982:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760776.234994:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27672576, 27676672) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.235008:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27672576=0x1a64000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.235065:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.235068:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27676672 00000020:00010000:1.0:1789760776.235142:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.235147:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.235151:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.235155:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.235158:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.235181:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.235188:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.235204:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760776.235309:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97100 - object 0x240000400:5 offset 27672576<>27676672 00000100:00100000:0.0:1789760776.235395:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512728064:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760776.286731:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974075 gen 3 00000008:00010000:0.0:1789760776.287620:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760776.287629:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.287640:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.287664:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760776.287698:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512728064:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760776.287996:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6756, 6756] 00000080:00200000:1.0:1789760776.288013:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.288019:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760776.288035:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27676672 i_blocks 55303 00000020:00200000:1.0:1789760776.288039:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760776.288050:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760776.288085:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27676672, count: 4096 00000080:00200000:1.0:1789760776.288093:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27676672, bytes: 4096 00000080:00200000:1.0:1789760776.288106:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760776.288116:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760776.288118:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27676672, 27680768) 0 0 00000080:00200000:1.0:1789760776.288123:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760776.288141:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6757, 6757] 00020000:00200000:1.0:1789760776.288151:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27676672 chunk: [27676672, 27680768) 27680768, 4096 00020000:00200000:1.0:1789760776.288156:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760776.288218:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6757, 6757] 00000008:00010000:1.0:1789760776.288305:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760776.288312:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760776.288325:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27676672, 27680768) append 0 append lockpos 0 00000080:00200000:1.0:1789760776.288339:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27676672=0x1a65000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760776.288402:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760776.288405:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27680768 00000020:00010000:1.0:1789760776.288415:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760776.288419:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.288423:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0f38/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760776.288426:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760776.288429:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760776.288437:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760776.288444:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760776.288578:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760776.288824:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 27676672<>27680768 00000100:00100000:0.0:1789760776.288982:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512728192:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760776.349822:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760776.349847:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.349859:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.349878:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760776.349935:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512728192:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760776.350220:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6757, 6757] 00000080:00200000:1.0:1789760776.350360:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760776.350368:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.350442:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27680768 i_blocks 55303 00000020:00200000:2.0:1789760776.350448:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.350461:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.350495:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27680768, count: 4096 00000080:00200000:2.0:1789760776.350502:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27680768, bytes: 4096 00000080:00200000:2.0:1789760776.350516:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.350525:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.350527:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27680768, 27684864) 0 0 00000080:00200000:2.0:1789760776.350531:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.350549:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6758, 6758] 00020000:00200000:2.0:1789760776.350559:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27680768 chunk: [27680768, 27684864) 27684864, 4096 00020000:00200000:2.0:1789760776.350564:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.350584:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6758, 6758] 00000008:00010000:2.0:1789760776.350603:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.350609:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760776.350621:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27680768, 27684864) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.350635:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27680768=0x1a66000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.350691:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.350694:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27684864 00000020:00010000:2.0:1789760776.350703:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.350706:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.350709:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.350713:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.350716:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.352105:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.352125:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.352211:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760776.352324:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 27680768<>27684864 00000100:00100000:3.0:1789760776.352426:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512728320:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760776.415191:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974077 gen 3 00000008:00010000:2.0:1789760776.415266:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:2.0:1789760776.415275:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760776.415285:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760776.415306:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:2.0:1789760776.415335:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512728320:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760776.415483:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6758, 6758] 00000080:00200000:3.0:1789760776.415499:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760776.415505:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760776.415519:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27684864 i_blocks 55303 00000020:00200000:3.0:1789760776.415523:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760776.415534:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760776.415578:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27684864, count: 4096 00000080:00200000:3.0:1789760776.415586:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27684864, bytes: 4096 00000080:00200000:3.0:1789760776.415596:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760776.415604:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760776.415606:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27684864, 27688960) 0 0 00000080:00200000:3.0:1789760776.415611:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760776.415627:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6759, 6759] 00020000:00200000:3.0:1789760776.415635:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27684864 chunk: [27684864, 27688960) 27688960, 4096 00020000:00200000:3.0:1789760776.415639:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760776.415658:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6759, 6759] 00000008:00010000:3.0:1789760776.415674:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760776.415691:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760776.415702:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27684864, 27688960) append 0 append lockpos 0 00000080:00200000:3.0:1789760776.415716:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27684864=0x1a67000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760776.415774:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760776.415777:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27688960 00000020:00010000:3.0:1789760776.415785:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760776.415788:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.415791:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.415795:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760776.415798:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760776.415805:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760776.415813:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760776.415829:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760776.416156:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 27684864<>27688960 00000100:00100000:2.0:1789760776.416240:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512728448:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760776.469262:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974078 gen 3 00000008:00010000:2.0:1789760776.469316:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760776.469325:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760776.469338:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760776.469360:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760776.469391:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512728448:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760776.469675:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6759, 6759] 00000080:00200000:3.0:1789760776.469772:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760776.469779:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760776.469862:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27688960 i_blocks 55303 00000020:00200000:3.0:1789760776.469868:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760776.469877:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760776.469944:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27688960, count: 4096 00000080:00200000:3.0:1789760776.469952:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27688960, bytes: 4096 00000080:00200000:3.0:1789760776.469964:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760776.469973:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760776.469976:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27688960, 27693056) 0 0 00000080:00200000:3.0:1789760776.469980:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760776.469998:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6760, 6760] 00020000:00200000:3.0:1789760776.470007:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27688960 chunk: [27688960, 27693056) 27693056, 4096 00020000:00200000:3.0:1789760776.470011:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760776.470031:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6760, 6760] 00000008:00010000:3.0:1789760776.470049:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760776.470056:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760776.470068:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27688960, 27693056) append 0 append lockpos 0 00000080:00200000:3.0:1789760776.470082:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27688960=0x1a68000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760776.470145:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760776.470148:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27693056 00000020:00010000:3.0:1789760776.470156:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760776.470159:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.470174:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.470178:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760776.470181:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760776.470189:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760776.470196:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760776.470216:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760776.470324:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adc00 - object 0x240000400:5 offset 27688960<>27693056 00000100:00100000:2.0:1789760776.470404:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512728576:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760776.528124:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974079 gen 3 00000008:00010000:2.0:1789760776.528187:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760776.528197:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760776.528210:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760776.528232:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760776.528263:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512728576:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760776.528371:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6760, 6760] 00000080:00200000:3.0:1789760776.528390:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760776.528396:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760776.528412:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27693056 i_blocks 55303 00000020:00200000:3.0:1789760776.528417:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760776.528427:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760776.528465:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27693056, count: 4096 00000080:00200000:3.0:1789760776.528473:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27693056, bytes: 4096 00000080:00200000:3.0:1789760776.528482:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760776.528492:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760776.528494:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27693056, 27697152) 0 0 00000080:00200000:3.0:1789760776.528499:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760776.528517:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6761, 6761] 00020000:00200000:3.0:1789760776.528526:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27693056 chunk: [27693056, 27697152) 27697152, 4096 00020000:00200000:3.0:1789760776.528530:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760776.528551:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6761, 6761] 00000008:00010000:3.0:1789760776.528581:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760776.528588:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760776.528601:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27693056, 27697152) append 0 append lockpos 0 00000080:00200000:3.0:1789760776.528614:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27693056=0x1a69000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760776.528677:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760776.528680:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27697152 00000020:00010000:3.0:1789760776.528689:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760776.528693:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.528696:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.528699:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760776.528702:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760776.528710:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760776.528717:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760776.528734:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760776.528837:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107afb80 - object 0x240000400:5 offset 27693056<>27697152 00000100:00100000:2.0:1789760776.528935:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512728704:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760776.608306:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974080 gen 3 00000008:00010000:2.0:1789760776.608365:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760776.608373:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760776.608385:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760776.608406:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760776.608436:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512728704:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760776.608789:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6761, 6761] 00000080:00200000:3.0:1789760776.608807:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760776.608813:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760776.608830:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27697152 i_blocks 55303 00000020:00200000:3.0:1789760776.608835:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760776.608845:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760776.608883:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27697152, count: 4096 00000080:00200000:3.0:1789760776.608891:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27697152, bytes: 4096 00000080:00200000:3.0:1789760776.608934:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760776.608945:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760776.608947:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27697152, 27701248) 0 0 00000080:00200000:3.0:1789760776.608952:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760776.608975:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6762, 6762] 00020000:00200000:3.0:1789760776.608984:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27697152 chunk: [27697152, 27701248) 27701248, 4096 00020000:00200000:3.0:1789760776.608989:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760776.609011:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6762, 6762] 00000008:00010000:3.0:1789760776.609030:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760776.609036:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760776.609047:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27697152, 27701248) append 0 append lockpos 0 00000080:00200000:3.0:1789760776.609062:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27697152=0x1a6a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760776.609126:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760776.609130:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27701248 00000020:00010000:3.0:1789760776.609137:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760776.609141:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.609144:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.609147:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760776.609150:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760776.609158:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760776.609173:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760776.609191:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760776.609301:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aca80 - object 0x240000400:5 offset 27697152<>27701248 00000100:00100000:2.0:1789760776.609593:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512728832:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760776.681566:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974081 gen 3 00000008:00010000:2.0:1789760776.681622:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760776.681631:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760776.681644:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760776.681731:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760776.681767:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512728832:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760776.682947:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6762, 6762] 00000080:00200000:3.0:1789760776.682969:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760776.682976:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760776.682993:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27701248 i_blocks 55303 00000020:00200000:3.0:1789760776.682999:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760776.683012:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760776.683043:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27701248, count: 4096 00000080:00200000:3.0:1789760776.683051:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27701248, bytes: 4096 00000080:00200000:3.0:1789760776.683065:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760776.683074:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760776.683077:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27701248, 27705344) 0 0 00000080:00200000:3.0:1789760776.683081:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760776.683157:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6763, 6763] 00020000:00200000:3.0:1789760776.683174:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27701248 chunk: [27701248, 27705344) 27705344, 4096 00020000:00200000:3.0:1789760776.683179:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760776.683209:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6763, 6763] 00000008:00010000:3.0:1789760776.683234:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760776.683240:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760776.683252:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27701248, 27705344) append 0 append lockpos 0 00000080:00200000:3.0:1789760776.683266:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27701248=0x1a6b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760776.683332:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760776.683335:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27705344 00000020:00010000:3.0:1789760776.683344:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760776.683347:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.683350:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.683353:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760776.683356:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760776.683365:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760776.683371:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760776.683397:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760776.683504:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac000 - object 0x240000400:5 offset 27701248<>27705344 00000100:00100000:3.0:1789760776.683560:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760776.683584:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512728960:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760776.734477:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760776.734493:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760776.734506:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760776.734576:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760776.734617:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512728960:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760776.735142:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6763, 6763] 00000080:00200000:3.0:1789760776.735168:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760776.735174:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760776.735189:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27705344 i_blocks 55303 00000020:00200000:3.0:1789760776.735193:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760776.735202:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760776.735239:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27705344, count: 4096 00000080:00200000:3.0:1789760776.735247:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27705344, bytes: 4096 00000080:00200000:3.0:1789760776.735258:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760776.735267:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760776.735270:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27705344, 27709440) 0 0 00000080:00200000:3.0:1789760776.735275:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760776.735292:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6764, 6764] 00020000:00200000:3.0:1789760776.735301:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27705344 chunk: [27705344, 27709440) 27709440, 4096 00020000:00200000:3.0:1789760776.735305:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760776.735324:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6764, 6764] 00000008:00010000:3.0:1789760776.735343:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760776.735349:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760776.735362:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27705344, 27709440) append 0 append lockpos 0 00000080:00200000:3.0:1789760776.735375:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27705344=0x1a6c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760776.735436:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760776.735439:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27709440 00000020:00010000:3.0:1789760776.735454:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760776.735458:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.735461:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760776.735465:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760776.735467:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760776.735476:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760776.735483:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760776.735505:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760776.735674:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad500 - object 0x240000400:5 offset 27705344<>27709440 00000100:00100000:2.0:1789760776.735746:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512729088:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760776.773098:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974083 gen 3 00000008:00010000:2.0:1789760776.773174:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760776.773182:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760776.773193:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760776.773221:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760776.773253:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512729088:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760776.773368:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6764, 6764] 00000080:00200000:0.0:1789760776.773385:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760776.773391:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760776.773406:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27709440 i_blocks 55303 00000020:00200000:0.0:1789760776.773411:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760776.773422:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760776.773457:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27709440, count: 4096 00000080:00200000:0.0:1789760776.773465:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27709440, bytes: 4096 00000080:00200000:0.0:1789760776.773476:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760776.773486:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760776.773488:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27709440, 27713536) 0 0 00000080:00200000:0.0:1789760776.773493:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760776.773512:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6765, 6765] 00020000:00200000:0.0:1789760776.773521:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27709440 chunk: [27709440, 27713536) 27713536, 4096 00020000:00200000:0.0:1789760776.773525:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760776.773545:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6765, 6765] 00000008:00010000:0.0:1789760776.773572:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760776.773579:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760776.773592:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27709440, 27713536) append 0 append lockpos 0 00000080:00200000:0.0:1789760776.773607:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27709440=0x1a6d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760776.773666:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760776.773669:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27713536 00000020:00010000:0.0:1789760776.773678:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760776.773682:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760776.773685:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760776.773688:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760776.773692:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760776.773700:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760776.773707:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760776.773724:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000008:00100000:0.0:1789760776.774068:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b480 - object 0x240000400:5 offset 27709440<>27713536 00000100:00100000:0.0:1789760776.774107:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760776.774125:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512729216:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760776.809082:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974084 gen 3 00000008:00010000:0.0:1789760776.809137:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:0.0:1789760776.809145:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760776.809157:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760776.809191:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:0.0:1789760776.809220:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512729216:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760776.809333:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6765, 6765] 00000080:00200000:2.0:1789760776.809350:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.809356:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.809370:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27713536 i_blocks 55303 00000020:00200000:2.0:1789760776.809374:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.809385:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.809419:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27713536, count: 4096 00000080:00200000:2.0:1789760776.809426:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27713536, bytes: 4096 00000080:00200000:2.0:1789760776.809438:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.809447:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.809450:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27713536, 27717632) 0 0 00000080:00200000:2.0:1789760776.809454:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.809471:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6766, 6766] 00020000:00200000:2.0:1789760776.809479:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27713536 chunk: [27713536, 27717632) 27717632, 4096 00020000:00200000:2.0:1789760776.809483:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.809502:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6766, 6766] 00000008:00010000:2.0:1789760776.809520:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.809526:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760776.809537:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27713536, 27717632) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.809561:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27713536=0x1a6e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.809619:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.809622:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27717632 00000020:00010000:2.0:1789760776.809630:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.809634:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.809637:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.809640:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.809643:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.809650:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.809656:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.809672:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760776.809774:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 27713536<>27717632 00000100:00100000:2.0:1789760776.809831:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760776.809849:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512729344:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760776.871773:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974085 gen 3 00000008:00010000:2.0:1789760776.871825:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:2.0:1789760776.871842:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760776.871853:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760776.871877:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:2.0:1789760776.871947:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512729344:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760776.872115:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6766, 6766] 00000080:00200000:0.0:1789760776.872132:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760776.872138:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760776.872154:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27717632 i_blocks 55303 00000020:00200000:0.0:1789760776.872157:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760776.872179:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760776.872215:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27717632, count: 4096 00000080:00200000:0.0:1789760776.872222:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27717632, bytes: 4096 00000080:00200000:0.0:1789760776.872233:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760776.872242:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760776.872245:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27717632, 27721728) 0 0 00000080:00200000:0.0:1789760776.872250:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760776.872269:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6767, 6767] 00020000:00200000:0.0:1789760776.872278:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27717632 chunk: [27717632, 27721728) 27721728, 4096 00020000:00200000:0.0:1789760776.872283:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760776.872305:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6767, 6767] 00000008:00010000:0.0:1789760776.872324:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760776.872330:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760776.872341:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27717632, 27721728) append 0 append lockpos 0 00000080:00200000:0.0:1789760776.872356:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27717632=0x1a6f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760776.872415:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760776.872418:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27721728 00000020:00010000:0.0:1789760776.872427:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760776.872430:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760776.872433:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760776.872437:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760776.872440:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760776.872449:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760776.872457:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760776.872476:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000008:00100000:0.0:1789760776.872601:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b480 - object 0x240000400:5 offset 27717632<>27721728 00000100:00100000:1.0:1789760776.872946:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760776.872975:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512729472:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760776.909756:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974086 gen 3 00000008:00010000:1.0:1789760776.909806:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760776.909814:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760776.909826:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760776.909852:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760776.909882:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512729472:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760776.910133:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6767, 6767] 00000080:00200000:0.0:1789760776.910149:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760776.910155:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760776.910181:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27721728 i_blocks 55303 00000020:00200000:0.0:1789760776.910185:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760776.910195:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760776.910229:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27721728, count: 4096 00000080:00200000:0.0:1789760776.910236:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27721728, bytes: 4096 00000080:00200000:0.0:1789760776.910246:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760776.910254:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760776.910257:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27721728, 27725824) 0 0 00000080:00200000:0.0:1789760776.910261:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760776.910278:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6768, 6768] 00020000:00200000:0.0:1789760776.910287:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27721728 chunk: [27721728, 27725824) 27725824, 4096 00020000:00200000:0.0:1789760776.910292:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760776.910310:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6768, 6768] 00000008:00010000:0.0:1789760776.910329:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760776.910334:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760776.910346:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27721728, 27725824) append 0 append lockpos 0 00000080:00200000:0.0:1789760776.910359:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27721728=0x1a70000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760776.910416:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760776.910419:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27725824 00000020:00010000:0.0:1789760776.910428:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760776.910431:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760776.910434:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760776.910438:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760776.910440:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760776.910448:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760776.910455:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760776.910472:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc900 00000008:00100000:0.0:1789760776.910590:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b800 - object 0x240000400:5 offset 27721728<>27725824 00000100:00100000:1.0:1789760776.910811:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760776.910841:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512729600:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760776.951355:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760776.951372:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760776.951387:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760776.951410:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760776.951441:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512729600:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760776.951613:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6768, 6768] 00000080:00200000:0.0:1789760776.951630:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760776.951637:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760776.951654:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27725824 i_blocks 55303 00000020:00200000:0.0:1789760776.951658:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760776.951668:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760776.951703:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27725824, count: 4096 00000080:00200000:0.0:1789760776.951710:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27725824, bytes: 4096 00000080:00200000:0.0:1789760776.951720:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760776.951729:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760776.951731:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27725824, 27729920) 0 0 00000080:00200000:0.0:1789760776.951736:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760776.951764:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6769, 6769] 00020000:00200000:0.0:1789760776.951772:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27725824 chunk: [27725824, 27729920) 27729920, 4096 00020000:00200000:0.0:1789760776.951777:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760776.951797:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6769, 6769] 00000008:00010000:0.0:1789760776.951815:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760776.951822:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760776.951832:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27725824, 27729920) append 0 append lockpos 0 00000080:00200000:0.0:1789760776.952218:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27725824=0x1a71000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760776.952286:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760776.952289:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27729920 00000020:00010000:0.0:1789760776.952297:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760776.952300:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760776.952303:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760776.952308:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760776.952311:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760776.952320:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760776.952328:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760776.952352:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cccc0 00000008:00100000:0.0:1789760776.952466:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148380 - object 0x240000400:5 offset 27725824<>27729920 00000100:00100000:1.0:1789760776.952543:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512729728:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760776.990217:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760776.996510:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974088 gen 3 00000008:00010000:1.0:1789760776.996576:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760776.996585:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760776.996596:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760776.996622:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760776.996652:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512729728:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760776.996755:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6769, 6769] 00000080:00200000:2.0:1789760776.996772:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760776.996778:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760776.996792:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27729920 i_blocks 55303 00000020:00200000:2.0:1789760776.996796:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760776.996806:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760776.996841:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27729920, count: 4096 00000080:00200000:2.0:1789760776.996849:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27729920, bytes: 4096 00000080:00200000:2.0:1789760776.996861:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760776.996869:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760776.996871:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27729920, 27734016) 0 0 00000080:00200000:2.0:1789760776.996875:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760776.996891:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6770, 6770] 00020000:00200000:2.0:1789760776.996938:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27729920 chunk: [27729920, 27734016) 27734016, 4096 00020000:00200000:2.0:1789760776.996943:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760776.996964:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6770, 6770] 00000008:00010000:2.0:1789760776.996981:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760776.996987:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760776.996998:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27729920, 27734016) append 0 append lockpos 0 00000080:00200000:2.0:1789760776.997012:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27729920=0x1a72000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760776.997077:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760776.997080:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27734016 00000020:00010000:2.0:1789760776.997089:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760776.997092:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.997095:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760776.997099:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760776.997102:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760776.997109:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760776.997116:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760776.997133:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760776.997243:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 27729920<>27734016 00000100:00100000:3.0:1789760776.997515:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512729856:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760777.037101:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974089 gen 3 00000008:00010000:2.0:1789760777.037145:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:2.0:1789760777.037152:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760777.037171:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760777.037192:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:2.0:1789760777.037215:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512729856:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760777.037329:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6770, 6770] 00000080:00200000:0.0:1789760777.037341:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760777.037346:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760777.037357:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27734016 i_blocks 55303 00000020:00200000:0.0:1789760777.037361:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760777.037369:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760777.037394:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27734016, count: 4096 00000080:00200000:0.0:1789760777.037401:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27734016, bytes: 4096 00000080:00200000:0.0:1789760777.037408:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760777.037416:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760777.037418:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27734016, 27738112) 0 0 00000080:00200000:0.0:1789760777.037422:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760777.037435:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6771, 6771] 00020000:00200000:0.0:1789760777.037442:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27734016 chunk: [27734016, 27738112) 27738112, 4096 00020000:00200000:0.0:1789760777.037446:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760777.038390:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6771, 6771] 00000008:00010000:0.0:1789760777.038409:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760777.038414:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760777.038424:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27734016, 27738112) append 0 append lockpos 0 00000080:00200000:0.0:1789760777.038436:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27734016=0x1a73000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760777.038475:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760777.038478:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27738112 00000020:00010000:0.0:1789760777.038484:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760777.038487:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.038490:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.038493:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760777.038495:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760777.038501:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760777.038509:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760777.038677:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000008:00100000:2.0:1789760777.038752:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 27734016<>27738112 00000100:00100000:3.0:1789760777.038817:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512729984:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760777.083231:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974090 gen 3 00000008:00010000:2.0:1789760777.083267:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:2.0:1789760777.083274:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760777.083283:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760777.083303:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:2.0:1789760777.083325:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512729984:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760777.083467:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6771, 6771] 00000080:00200000:0.0:1789760777.083479:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760777.083485:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760777.083496:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27738112 i_blocks 55303 00000020:00200000:0.0:1789760777.083500:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760777.083515:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760777.083540:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27738112, count: 4096 00000080:00200000:0.0:1789760777.083547:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27738112, bytes: 4096 00000080:00200000:0.0:1789760777.083615:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760777.083624:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760777.083626:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27738112, 27742208) 0 0 00000080:00200000:0.0:1789760777.083631:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760777.083644:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6772, 6772] 00020000:00200000:0.0:1789760777.083651:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27738112 chunk: [27738112, 27742208) 27742208, 4096 00020000:00200000:0.0:1789760777.083655:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760777.083669:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6772, 6772] 00000008:00010000:0.0:1789760777.083684:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760777.083689:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760777.083697:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27738112, 27742208) append 0 append lockpos 0 00000080:00200000:0.0:1789760777.083708:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27738112=0x1a74000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760777.083751:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760777.083754:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27742208 00000020:00010000:0.0:1789760777.083761:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760777.083764:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.083767:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.083771:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760777.083774:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760777.083780:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760777.083786:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760777.083801:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc600 00000008:00100000:0.0:1789760777.083881:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149880 - object 0x240000400:5 offset 27738112<>27742208 00000100:00100000:1.0:1789760777.083962:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512730112:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760777.148188:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760777.148200:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760777.148211:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760777.148238:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760777.148260:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512730112:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760777.148502:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6772, 6772] 00000080:00200000:0.0:1789760777.148513:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760777.148518:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760777.148530:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27742208 i_blocks 55303 00000020:00200000:0.0:1789760777.148534:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760777.148542:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760777.148566:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27742208, count: 4096 00000080:00200000:0.0:1789760777.148572:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27742208, bytes: 4096 00000080:00200000:0.0:1789760777.148580:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760777.148587:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760777.148590:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27742208, 27746304) 0 0 00000080:00200000:0.0:1789760777.148593:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760777.148606:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6773, 6773] 00020000:00200000:0.0:1789760777.148613:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27742208 chunk: [27742208, 27746304) 27746304, 4096 00020000:00200000:0.0:1789760777.148616:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760777.148631:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6773, 6773] 00000008:00010000:0.0:1789760777.148646:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760777.148651:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760777.148661:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27742208, 27746304) append 0 append lockpos 0 00000080:00200000:0.0:1789760777.148672:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27742208=0x1a75000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760777.148988:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760777.148991:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27746304 00000020:00010000:0.0:1789760777.148999:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760777.149003:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.149006:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.149009:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760777.149011:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760777.149017:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760777.149022:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760777.149045:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc540 00000008:00100000:0.0:1789760777.149120:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a300 - object 0x240000400:5 offset 27742208<>27746304 00000100:00100000:1.0:1789760777.149436:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512730240:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760777.182128:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974092 gen 3 00000008:00010000:1.0:1789760777.182185:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760777.182192:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760777.182201:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760777.182212:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760777.182234:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512730240:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760777.183428:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6773, 6773] 00000080:00200000:0.0:1789760777.183442:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760777.183448:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760777.183473:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27746304 i_blocks 55303 00000020:00200000:0.0:1789760777.183478:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760777.183489:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760777.183513:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27746304, count: 4096 00000080:00200000:0.0:1789760777.183520:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27746304, bytes: 4096 00000080:00200000:0.0:1789760777.183528:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760777.183535:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760777.183537:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27746304, 27750400) 0 0 00000080:00200000:0.0:1789760777.183541:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760777.183553:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6774, 6774] 00020000:00200000:0.0:1789760777.183559:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27746304 chunk: [27746304, 27750400) 27750400, 4096 00020000:00200000:0.0:1789760777.183563:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760777.183580:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6774, 6774] 00000008:00010000:0.0:1789760777.183594:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760777.183599:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760777.183614:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27746304, 27750400) append 0 append lockpos 0 00000080:00200000:0.0:1789760777.183626:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27746304=0x1a76000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760777.183675:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760777.183678:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27750400 00000020:00010000:0.0:1789760777.183685:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760777.183688:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.183691:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.183693:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760777.183696:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760777.183701:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760777.183707:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760777.183721:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cca80 00000008:00100000:0.0:1789760777.183796:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a680 - object 0x240000400:5 offset 27746304<>27750400 00000100:00100000:1.0:1789760777.183856:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512730368:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760777.217967:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974093 gen 3 00000008:00010000:1.0:1789760777.218004:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760777.218011:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760777.218020:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760777.218043:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760777.218066:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512730368:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760777.218863:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6774, 6774] 00000080:00200000:0.0:1789760777.218876:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760777.218881:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760777.218892:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27750400 i_blocks 55303 00000020:00200000:0.0:1789760777.218896:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760777.218933:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760777.218959:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27750400, count: 4096 00000080:00200000:0.0:1789760777.218965:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27750400, bytes: 4096 00000080:00200000:0.0:1789760777.218974:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760777.218981:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760777.218983:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27750400, 27754496) 0 0 00000080:00200000:0.0:1789760777.218987:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760777.218999:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6775, 6775] 00020000:00200000:0.0:1789760777.219005:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27750400 chunk: [27750400, 27754496) 27754496, 4096 00020000:00200000:0.0:1789760777.219008:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760777.219024:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6775, 6775] 00000008:00010000:0.0:1789760777.219037:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760777.219041:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760777.219049:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27750400, 27754496) append 0 append lockpos 0 00000080:00200000:0.0:1789760777.219059:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27750400=0x1a77000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760777.219109:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760777.219112:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27754496 00000020:00010000:0.0:1789760777.219118:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760777.219121:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.219124:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.219127:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760777.219129:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760777.219136:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760777.219141:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760777.219165:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccf00 00000008:00100000:0.0:1789760777.219241:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148e00 - object 0x240000400:5 offset 27750400<>27754496 00000100:00100000:1.0:1789760777.219654:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512730496:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760777.264779:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760777.264894:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760777.264922:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760777.265011:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760777.265035:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512730496:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760777.265098:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6775, 6775] 00000080:00200000:0.0:1789760777.265111:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760777.265116:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760777.265204:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27754496 i_blocks 55303 00000020:00200000:0.0:1789760777.265208:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760777.265216:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760777.265241:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27754496, count: 4096 00000080:00200000:0.0:1789760777.265247:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27754496, bytes: 4096 00000080:00200000:0.0:1789760777.265256:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760777.265262:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760777.265265:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27754496, 27758592) 0 0 00000080:00200000:0.0:1789760777.265268:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760777.265283:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6776, 6776] 00020000:00200000:0.0:1789760777.265288:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27754496 chunk: [27754496, 27758592) 27758592, 4096 00020000:00200000:0.0:1789760777.265292:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760777.265306:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6776, 6776] 00000008:00010000:0.0:1789760777.265319:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760777.265323:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760777.265332:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27754496, 27758592) append 0 append lockpos 0 00000080:00200000:0.0:1789760777.265342:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27754496=0x1a78000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760777.265384:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760777.265387:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27758592 00000020:00010000:0.0:1789760777.265393:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760777.265396:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.265399:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.265402:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760777.265405:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760777.265411:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760777.265416:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760777.265430:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc000 00000008:00100000:0.0:1789760777.265505:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148a80 - object 0x240000400:5 offset 27754496<>27758592 00000100:00100000:1.0:1789760777.265572:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512730624:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760777.306413:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974094 gen 3 00000100:00100000:1.0:1789760777.306444:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005148a80 x1876699512730624/t4294974095(4294974095) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760793 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760777.306489:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760777.306494:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760777.306503:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760777.306522:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760777.306544:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512730624:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760777.306688:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6776, 6776] 00000080:00200000:0.0:1789760777.306701:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760777.306706:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760777.306718:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27758592 i_blocks 55303 00000020:00200000:0.0:1789760777.306722:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760777.306730:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760777.306754:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27758592, count: 4096 00000080:00200000:0.0:1789760777.306760:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27758592, bytes: 4096 00000080:00200000:0.0:1789760777.306769:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760777.306776:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760777.306778:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27758592, 27762688) 0 0 00000080:00200000:0.0:1789760777.306781:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760777.306796:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6777, 6777] 00020000:00200000:0.0:1789760777.306803:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27758592 chunk: [27758592, 27762688) 27762688, 4096 00020000:00200000:0.0:1789760777.306807:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760777.306822:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6777, 6777] 00000008:00010000:0.0:1789760777.306836:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760777.306840:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760777.306850:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27758592, 27762688) append 0 append lockpos 0 00000080:00200000:0.0:1789760777.306862:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27758592=0x1a79000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760777.306948:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760777.306951:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27762688 00000020:00010000:0.0:1789760777.306958:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760777.306961:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.306964:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.306967:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760777.306970:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760777.306977:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760777.306982:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760777.306995:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000008:00100000:0.0:1789760777.307069:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514ad80 - object 0x240000400:5 offset 27758592<>27762688 00000100:00100000:1.0:1789760777.309353:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512730752:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760777.351523:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974096 gen 3 00000008:00010000:1.0:1789760777.351676:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760777.351684:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760777.351692:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760777.351711:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760777.351733:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512730752:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760777.352658:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6777, 6777] 00000080:00200000:0.0:1789760777.352671:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760777.352676:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760777.352688:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27762688 i_blocks 55303 00000020:00200000:0.0:1789760777.352692:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760777.352701:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760777.352725:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27762688, count: 4096 00000080:00200000:0.0:1789760777.352731:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27762688, bytes: 4096 00000080:00200000:0.0:1789760777.352739:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760777.352746:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760777.352748:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27762688, 27766784) 0 0 00000080:00200000:0.0:1789760777.352752:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760777.352765:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6778, 6778] 00020000:00200000:0.0:1789760777.352772:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27762688 chunk: [27762688, 27766784) 27766784, 4096 00020000:00200000:0.0:1789760777.352776:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760777.352790:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6778, 6778] 00000008:00010000:0.0:1789760777.352803:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760777.352808:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760777.352817:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27762688, 27766784) append 0 append lockpos 0 00000080:00200000:0.0:1789760777.352828:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27762688=0x1a7a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760777.352874:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760777.352876:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27766784 00000020:00010000:0.0:1789760777.352882:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760777.352886:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.352889:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.352891:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760777.352894:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760777.352941:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760777.352948:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760777.352964:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc780 00000008:00100000:0.0:1789760777.353041:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148700 - object 0x240000400:5 offset 27762688<>27766784 00000100:00100000:0.0:1789760777.353074:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760777.353089:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512730880:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760777.410897:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974097 gen 3 00000008:00010000:0.0:1789760777.410984:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:0.0:1789760777.410994:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.411006:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.411094:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:0.0:1789760777.411126:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512730880:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760777.411221:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6778, 6778] 00000080:00200000:2.0:1789760777.411238:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760777.411244:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760777.411258:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27766784 i_blocks 55303 00000020:00200000:2.0:1789760777.411262:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760777.411274:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760777.411310:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27766784, count: 4096 00000080:00200000:2.0:1789760777.411319:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27766784, bytes: 4096 00000080:00200000:2.0:1789760777.411331:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760777.411341:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760777.411343:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27766784, 27770880) 0 0 00000080:00200000:2.0:1789760777.411347:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760777.411364:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6779, 6779] 00020000:00200000:2.0:1789760777.411372:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27766784 chunk: [27766784, 27770880) 27770880, 4096 00020000:00200000:2.0:1789760777.411377:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760777.411396:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6779, 6779] 00000008:00010000:2.0:1789760777.411416:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760777.411422:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760777.411433:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27766784, 27770880) append 0 append lockpos 0 00000080:00200000:2.0:1789760777.411447:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27766784=0x1a7b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760777.411507:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760777.411510:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27770880 00000020:00010000:2.0:1789760777.411519:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760777.411523:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.411526:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.411529:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760777.411532:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760777.411540:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760777.411547:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760777.411565:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760777.411668:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 27766784<>27770880 00000100:00100000:3.0:1789760777.411740:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512731008:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760777.449527:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974098 gen 3 00000008:00010000:2.0:1789760777.449628:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:2.0:1789760777.449635:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760777.449644:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760777.449726:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:2.0:1789760777.449751:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512731008:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760777.450137:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6779, 6779] 00000080:00200000:0.0:1789760777.450222:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760777.450227:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760777.450239:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27770880 i_blocks 55303 00000020:00200000:0.0:1789760777.450243:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760777.450251:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760777.450281:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27770880, count: 4096 00000080:00200000:0.0:1789760777.450287:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27770880, bytes: 4096 00000080:00200000:0.0:1789760777.450297:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760777.450304:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760777.450306:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27770880, 27774976) 0 0 00000080:00200000:0.0:1789760777.450310:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760777.450325:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6780, 6780] 00020000:00200000:0.0:1789760777.450332:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27770880 chunk: [27770880, 27774976) 27774976, 4096 00020000:00200000:0.0:1789760777.450336:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760777.450352:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6780, 6780] 00000008:00010000:0.0:1789760777.450365:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760777.450370:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760777.450379:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27770880, 27774976) append 0 append lockpos 0 00000080:00200000:0.0:1789760777.450389:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27770880=0x1a7c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760777.450434:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760777.450437:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27774976 00000020:00010000:0.0:1789760777.450444:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760777.450447:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.450449:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.450452:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760777.450455:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760777.450462:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760777.450467:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760777.450483:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc780 00000008:00100000:0.0:1789760777.450637:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148700 - object 0x240000400:5 offset 27770880<>27774976 00000100:00100000:1.0:1789760777.450745:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512731136:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760777.481605:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974099 gen 3 00000008:00010000:1.0:1789760777.481643:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760777.481651:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760777.481661:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760777.481743:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760777.481769:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512731136:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760777.481846:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6780, 6780] 00000080:00200000:0.0:1789760777.481872:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760777.481878:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760777.481891:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27774976 i_blocks 55303 00000020:00200000:0.0:1789760777.481894:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760777.481931:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760777.481959:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27774976, count: 4096 00000080:00200000:0.0:1789760777.481964:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27774976, bytes: 4096 00000080:00200000:0.0:1789760777.481972:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760777.481980:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760777.481982:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27774976, 27779072) 0 0 00000080:00200000:0.0:1789760777.481986:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760777.482000:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6781, 6781] 00020000:00200000:0.0:1789760777.482006:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27774976 chunk: [27774976, 27779072) 27779072, 4096 00020000:00200000:0.0:1789760777.482010:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760777.482026:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6781, 6781] 00000008:00010000:0.0:1789760777.482041:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760777.482046:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760777.482056:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27774976, 27779072) append 0 append lockpos 0 00000080:00200000:0.0:1789760777.482066:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27774976=0x1a7d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760777.482111:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760777.482113:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27779072 00000020:00010000:0.0:1789760777.482119:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760777.482122:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.482125:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.482127:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760777.482130:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760777.482136:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760777.482141:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760777.482169:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccd80 00000008:00100000:0.0:1789760777.482247:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514aa00 - object 0x240000400:5 offset 27774976<>27779072 00000100:00100000:1.0:1789760777.482308:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512731264:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760777.519685:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974100 gen 3 00000008:00010000:1.0:1789760777.519722:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760777.519728:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760777.519737:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760777.519819:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760777.519843:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512731264:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760777.523038:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6781, 6781] 00000080:00200000:0.0:1789760777.523058:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760777.523063:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760777.523076:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27779072 i_blocks 55303 00000020:00200000:0.0:1789760777.523081:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760777.523093:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760777.523118:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27779072, count: 4096 00000080:00200000:0.0:1789760777.523125:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27779072, bytes: 4096 00000080:00200000:0.0:1789760777.523135:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760777.523142:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760777.523144:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27779072, 27783168) 0 0 00000080:00200000:0.0:1789760777.523147:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760777.523172:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6782, 6782] 00020000:00200000:0.0:1789760777.523179:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27779072 chunk: [27779072, 27783168) 27783168, 4096 00020000:00200000:0.0:1789760777.523183:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760777.523199:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6782, 6782] 00000008:00010000:0.0:1789760777.523214:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760777.523219:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760777.523229:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27779072, 27783168) append 0 append lockpos 0 00000080:00200000:0.0:1789760777.523895:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27779072=0x1a7e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760777.525888:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760777.525893:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27783168 00000020:00010000:0.0:1789760777.525921:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760777.525927:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.525930:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760777.525934:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760777.525936:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760777.525946:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760777.525954:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760777.525973:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc180 00000008:00100000:0.0:1789760777.526086:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514aa00 - object 0x240000400:5 offset 27779072<>27783168 00000100:00100000:0.0:1789760777.526144:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512731392:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760777.568263:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974101 gen 3 00000008:00010000:0.0:1789760777.568301:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:0.0:1789760777.568308:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.568317:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.568337:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000080:00200000:1.0:1789760777.568459:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6782, 6782] 00000080:00200000:1.0:1789760777.568474:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.568480:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.568491:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27783168 i_blocks 55303 00000020:00200000:1.0:1789760777.568494:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.568503:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.568527:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27783168, count: 4096 00000080:00200000:1.0:1789760777.568534:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27783168, bytes: 4096 00000080:00200000:1.0:1789760777.568541:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.568549:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.568551:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27783168, 27787264) 0 0 00000080:00200000:1.0:1789760777.568555:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.568568:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6783, 6783] 00020000:00200000:1.0:1789760777.568575:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27783168 chunk: [27783168, 27787264) 27787264, 4096 00020000:00200000:1.0:1789760777.568579:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.568602:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6783, 6783] 00000008:00010000:1.0:1789760777.568617:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.568623:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760777.568634:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27783168, 27787264) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.568644:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27783168=0x1a7f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.568682:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.568685:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27787264 00000020:00010000:1.0:1789760777.568692:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.568696:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.568698:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.568702:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.568715:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.568722:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.568727:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:0.0:1789760777.568771:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512731392:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760777.568806:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:0.0:1789760777.568871:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514aa00 - object 0x240000400:5 offset 27783168<>27787264 00000100:00100000:0.0:1789760777.568896:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760777.572513:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512731520:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760777.623230:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974102 gen 3 00000008:00010000:1.0:1789760777.623265:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760777.623272:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760777.623281:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760777.623308:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760777.623332:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512731520:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760777.623359:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6783, 6783] 00000080:00200000:2.0:1789760777.623372:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760777.623378:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760777.623390:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27787264 i_blocks 55303 00000020:00200000:2.0:1789760777.623394:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760777.623403:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760777.623429:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27787264, count: 4096 00000080:00200000:2.0:1789760777.623435:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27787264, bytes: 4096 00000080:00200000:2.0:1789760777.623443:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760777.623450:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760777.623452:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27787264, 27791360) 0 0 00000080:00200000:2.0:1789760777.623456:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760777.623470:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6784, 6784] 00020000:00200000:2.0:1789760777.623476:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27787264 chunk: [27787264, 27791360) 27791360, 4096 00020000:00200000:2.0:1789760777.623479:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760777.623495:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6784, 6784] 00000008:00010000:2.0:1789760777.623510:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760777.623514:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760777.623523:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27787264, 27791360) append 0 append lockpos 0 00000080:00200000:2.0:1789760777.623533:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27787264=0x1a80000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760777.623577:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760777.623580:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27791360 00000020:00010000:2.0:1789760777.623587:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760777.623590:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.623593:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.623596:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760777.623598:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760777.623605:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760777.623611:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760777.623625:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760777.623704:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 27787264<>27791360 00000100:00100000:2.0:1789760777.623748:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760777.623762:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512731648:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760777.681165:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974103 gen 3 00000008:00010000:2.0:1789760777.681207:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:2.0:1789760777.681214:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760777.681223:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760777.681249:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:2.0:1789760777.681274:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512731648:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760777.681429:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6784, 6784] 00000080:00200000:1.0:1789760777.681441:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.681446:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.681458:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27791360 i_blocks 55303 00000020:00200000:1.0:1789760777.681461:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.681471:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.681495:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27791360, count: 4096 00000080:00200000:1.0:1789760777.681501:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27791360, bytes: 4096 00000080:00200000:1.0:1789760777.681511:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.681519:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.681521:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27791360, 27795456) 0 0 00000080:00200000:1.0:1789760777.681524:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.681536:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6785, 6785] 00020000:00200000:1.0:1789760777.681542:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27791360 chunk: [27791360, 27795456) 27795456, 4096 00020000:00200000:1.0:1789760777.681546:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.681560:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6785, 6785] 00000008:00010000:1.0:1789760777.681573:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.681577:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760777.681586:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27791360, 27795456) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.681595:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27791360=0x1a81000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.681634:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.681637:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27795456 00000020:00010000:1.0:1789760777.681642:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.681645:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.681648:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.681651:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.681654:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.681660:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.681665:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760777.681678:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760777.681746:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 27791360<>27795456 00000100:00100000:1.0:1789760777.681789:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760777.681803:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512731776:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760777.735228:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760777.735239:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760777.735246:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760777.735275:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760777.735298:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512731776:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760777.735710:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6785, 6785] 00000080:00200000:2.0:1789760777.735721:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760777.735726:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760777.735736:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27795456 i_blocks 55303 00000020:00200000:2.0:1789760777.735739:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760777.735747:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760777.735771:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27795456, count: 4096 00000080:00200000:2.0:1789760777.735777:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27795456, bytes: 4096 00000080:00200000:2.0:1789760777.735787:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760777.735804:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760777.735806:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27795456, 27799552) 0 0 00000080:00200000:2.0:1789760777.735810:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760777.735821:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6786, 6786] 00020000:00200000:2.0:1789760777.735827:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27795456 chunk: [27795456, 27799552) 27799552, 4096 00020000:00200000:2.0:1789760777.735831:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760777.735845:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6786, 6786] 00000008:00010000:2.0:1789760777.735858:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760777.735863:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760777.735872:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27795456, 27799552) append 0 append lockpos 0 00000080:00200000:2.0:1789760777.735882:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27795456=0x1a82000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760777.735958:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760777.735960:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27799552 00000020:00010000:2.0:1789760777.735967:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760777.735970:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.735973:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.735976:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760777.735979:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760777.735985:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760777.735989:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760777.736002:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760777.736077:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 27795456<>27799552 00000100:00100000:3.0:1789760777.737881:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512731904:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760777.771713:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974105 gen 3 00000008:00010000:2.0:1789760777.771759:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:2.0:1789760777.771765:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760777.771773:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760777.772190:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:2.0:1789760777.772215:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512731904:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760777.772254:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6786, 6786] 00000080:00200000:1.0:1789760777.772377:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.772383:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.772394:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27799552 i_blocks 55303 00000020:00200000:1.0:1789760777.772397:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.772406:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.772429:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27799552, count: 4096 00000080:00200000:1.0:1789760777.772436:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27799552, bytes: 4096 00000080:00200000:1.0:1789760777.772445:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.772451:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.772454:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27799552, 27803648) 0 0 00000080:00200000:1.0:1789760777.772457:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.772467:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6787, 6787] 00020000:00200000:1.0:1789760777.772473:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27799552 chunk: [27799552, 27803648) 27803648, 4096 00020000:00200000:1.0:1789760777.772475:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.772487:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6787, 6787] 00000008:00010000:1.0:1789760777.772498:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.772501:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760777.772508:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27799552, 27803648) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.772517:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27799552=0x1a83000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.772561:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.772563:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27803648 00000020:00010000:1.0:1789760777.772568:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.772571:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.772573:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.772575:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.772577:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.772582:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.772587:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760777.772601:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760777.772689:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 27799552<>27803648 00000100:00100000:0.0:1789760777.772765:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760777.772784:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512732032:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760777.826010:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974106 gen 3 00000008:00010000:0.0:1789760777.826042:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760777.826048:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760777.826055:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760777.826075:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760777.826097:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512732032:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760777.826321:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6787, 6787] 00000080:00200000:2.0:1789760777.826334:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760777.826340:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760777.826352:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27803648 i_blocks 55303 00000020:00200000:2.0:1789760777.826356:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760777.826364:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760777.826395:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27803648, count: 4096 00000080:00200000:2.0:1789760777.826400:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27803648, bytes: 4096 00000080:00200000:2.0:1789760777.826409:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760777.826519:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760777.826522:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27803648, 27807744) 0 0 00000080:00200000:2.0:1789760777.826526:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760777.826539:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6788, 6788] 00020000:00200000:2.0:1789760777.826592:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27803648 chunk: [27803648, 27807744) 27807744, 4096 00020000:00200000:2.0:1789760777.826595:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760777.826609:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6788, 6788] 00000008:00010000:2.0:1789760777.826624:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760777.826628:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760777.826637:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27803648, 27807744) append 0 append lockpos 0 00000080:00200000:2.0:1789760777.826648:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27803648=0x1a84000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760777.826690:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760777.826692:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27807744 00000020:00010000:2.0:1789760777.826698:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760777.826701:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.826703:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760777.826705:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760777.826707:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760777.826712:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760777.826717:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760777.826732:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760777.827186:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 27803648<>27807744 00000100:00100000:3.0:1789760777.827242:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512732160:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760777.867795:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:2.0:1789760777.867806:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760777.867816:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760777.867871:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:2.0:1789760777.867895:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512732160:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760777.868043:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6788, 6788] 00000080:00200000:3.0:1789760777.868056:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760777.868061:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760777.868073:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27807744 i_blocks 55303 00000020:00200000:3.0:1789760777.868077:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760777.868173:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760777.868199:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27807744, count: 4096 00000080:00200000:3.0:1789760777.868205:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27807744, bytes: 4096 00000080:00200000:3.0:1789760777.868213:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760777.868220:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760777.868222:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27807744, 27811840) 0 0 00000080:00200000:3.0:1789760777.868226:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760777.868238:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6789, 6789] 00020000:00200000:3.0:1789760777.868244:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27807744 chunk: [27807744, 27811840) 27811840, 4096 00020000:00200000:3.0:1789760777.868248:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760777.868261:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6789, 6789] 00000008:00010000:3.0:1789760777.868276:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760777.868281:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760777.868290:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27807744, 27811840) append 0 append lockpos 0 00000080:00200000:3.0:1789760777.868301:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27807744=0x1a85000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760777.868350:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760777.868353:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27811840 00000020:00010000:3.0:1789760777.868360:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760777.868363:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.868365:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.868368:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760777.868370:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760777.868377:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760777.868383:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760777.868398:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760777.868475:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac700 - object 0x240000400:5 offset 27807744<>27811840 00000100:00100000:2.0:1789760777.868534:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512732288:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760777.901321:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974107 gen 3 00000100:00100000:2.0:1789760777.901361:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc0107ac700 x1876699512732288/t4294974108(4294974108) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760793 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:2.0:1789760777.901399:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760777.901405:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760777.901413:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760777.901426:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:2.0:1789760777.901446:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512732288:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760777.901931:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6789, 6789] 00000080:00200000:3.0:1789760777.901943:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760777.901947:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760777.901958:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27811840 i_blocks 55303 00000020:00200000:3.0:1789760777.901962:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760777.901969:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760777.901996:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27811840, count: 4096 00000080:00200000:3.0:1789760777.902001:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27811840, bytes: 4096 00000080:00200000:3.0:1789760777.902010:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760777.902017:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760777.902019:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27811840, 27815936) 0 0 00000080:00200000:3.0:1789760777.902023:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760777.902036:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6790, 6790] 00020000:00200000:3.0:1789760777.902043:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27811840 chunk: [27811840, 27815936) 27815936, 4096 00020000:00200000:3.0:1789760777.902046:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760777.902062:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6790, 6790] 00000008:00010000:3.0:1789760777.902075:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760777.902080:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760777.902321:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27811840, 27815936) append 0 append lockpos 0 00000080:00200000:3.0:1789760777.902333:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27811840=0x1a86000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760777.902382:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760777.902385:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27815936 00000020:00010000:3.0:1789760777.902392:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760777.902396:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.902399:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.902402:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760777.902405:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760777.902411:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760777.902417:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760777.902434:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760777.902519:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae300 - object 0x240000400:5 offset 27811840<>27815936 00000100:00100000:2.0:1789760777.906442:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760777.906471:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512732416:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760777.948346:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974109 gen 3 00000008:00010000:2.0:1789760777.948391:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760777.948397:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760777.948406:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760777.948423:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760777.948443:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512732416:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760777.948743:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6790, 6790] 00000080:00200000:3.0:1789760777.948756:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760777.948762:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760777.948776:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27815936 i_blocks 55303 00000020:00200000:3.0:1789760777.948780:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760777.948797:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760777.948832:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27815936, count: 4096 00000080:00200000:3.0:1789760777.948839:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27815936, bytes: 4096 00000080:00200000:3.0:1789760777.948847:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760777.948854:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760777.948857:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27815936, 27820032) 0 0 00000080:00200000:3.0:1789760777.948861:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760777.948873:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6791, 6791] 00020000:00200000:3.0:1789760777.948880:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27815936 chunk: [27815936, 27820032) 27820032, 4096 00020000:00200000:3.0:1789760777.948883:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760777.948934:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6791, 6791] 00000008:00010000:3.0:1789760777.948959:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760777.948964:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760777.948973:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27815936, 27820032) append 0 append lockpos 0 00000080:00200000:3.0:1789760777.948985:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27815936=0x1a87000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760777.949031:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760777.949034:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27820032 00000020:00010000:3.0:1789760777.949040:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760777.949044:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.949046:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760777.949049:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760777.949051:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760777.949057:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760777.949062:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760777.949077:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760777.949169:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af480 - object 0x240000400:5 offset 27815936<>27820032 00000100:00100000:3.0:1789760777.949208:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760777.949221:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512732544:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760777.988078:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974110 gen 3 00000008:00010000:2.0:1789760777.988111:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760777.988117:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760777.988126:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760777.988206:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760777.988228:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512732544:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760777.988289:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6791, 6791] 00000080:00200000:1.0:1789760777.988301:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760777.988306:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760777.988316:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27820032 i_blocks 55303 00000020:00200000:1.0:1789760777.988319:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760777.988326:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760777.988350:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27820032, count: 4096 00000080:00200000:1.0:1789760777.988356:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27820032, bytes: 4096 00000080:00200000:1.0:1789760777.988364:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760777.988370:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760777.988372:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27820032, 27824128) 0 0 00000080:00200000:1.0:1789760777.988376:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760777.988387:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6792, 6792] 00020000:00200000:1.0:1789760777.988394:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27820032 chunk: [27820032, 27824128) 27824128, 4096 00020000:00200000:1.0:1789760777.988397:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760777.988411:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6792, 6792] 00000008:00010000:1.0:1789760777.988424:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760777.988428:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760777.988437:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27820032, 27824128) append 0 append lockpos 0 00000080:00200000:1.0:1789760777.988447:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27820032=0x1a88000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760777.988488:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760777.988490:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27824128 00000020:00010000:1.0:1789760777.988496:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760777.988499:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.988502:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760777.988504:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760777.988506:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760777.988512:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760777.988517:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760777.988531:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760777.988666:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95c00 - object 0x240000400:5 offset 27820032<>27824128 00000100:00100000:0.0:1789760777.988728:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512732672:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760778.013877:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760778.019213:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974111 gen 3 00000008:00010000:0.0:1789760778.019249:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760778.019255:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.019263:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.019284:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760778.019307:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512732672:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760778.019404:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6792, 6792] 00000080:00200000:1.0:1789760778.019416:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.019421:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.019432:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27824128 i_blocks 55303 00000020:00200000:1.0:1789760778.019435:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.019443:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.019467:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27824128, count: 4096 00000080:00200000:1.0:1789760778.019472:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27824128, bytes: 4096 00000080:00200000:1.0:1789760778.019479:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.019486:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.019488:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27824128, 27828224) 0 0 00000080:00200000:1.0:1789760778.019492:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.019505:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6793, 6793] 00020000:00200000:1.0:1789760778.019512:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27824128 chunk: [27824128, 27828224) 27828224, 4096 00020000:00200000:1.0:1789760778.019515:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.019529:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6793, 6793] 00000008:00010000:1.0:1789760778.019553:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.019557:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760778.019567:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27824128, 27828224) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.019578:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27824128=0x1a89000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.019620:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.019623:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27828224 00000020:00010000:1.0:1789760778.019629:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.019632:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.019634:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.019637:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.019640:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.019646:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.019650:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.019663:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760778.019729:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 27824128<>27828224 00000100:00100000:0.0:1789760778.019934:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760778.019947:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512732800:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760778.049184:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974112 gen 3 00000008:00010000:0.0:1789760778.049220:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760778.049226:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.049235:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.049254:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760778.049278:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512732800:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760778.049372:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6793, 6793] 00000080:00200000:1.0:1789760778.049385:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.049390:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.049403:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27828224 i_blocks 55303 00000020:00200000:1.0:1789760778.049406:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.049413:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.049439:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27828224, count: 4096 00000080:00200000:1.0:1789760778.049445:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27828224, bytes: 4096 00000080:00200000:1.0:1789760778.049452:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.049459:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.049461:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27828224, 27832320) 0 0 00000080:00200000:1.0:1789760778.049465:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.049477:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6794, 6794] 00020000:00200000:1.0:1789760778.049484:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27828224 chunk: [27828224, 27832320) 27832320, 4096 00020000:00200000:1.0:1789760778.049488:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.049502:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6794, 6794] 00000008:00010000:1.0:1789760778.049515:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.049520:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760778.049528:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27828224, 27832320) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.049537:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27828224=0x1a8a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.049594:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.049596:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27832320 00000020:00010000:1.0:1789760778.049602:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.049605:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.049608:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.049611:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.049614:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.049620:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.049626:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.049639:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760778.049715:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 27828224<>27832320 00000100:00100000:0.0:1789760778.050007:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760778.050021:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512732928:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760778.084290:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974113 gen 3 00000008:00010000:0.0:1789760778.084330:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760778.084338:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.084347:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.084365:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760778.084388:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512732928:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760778.084478:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6794, 6794] 00000080:00200000:1.0:1789760778.084492:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.084497:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.084510:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27832320 i_blocks 55303 00000020:00200000:1.0:1789760778.084513:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.084522:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.084557:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27832320, count: 4096 00000080:00200000:1.0:1789760778.084564:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27832320, bytes: 4096 00000080:00200000:1.0:1789760778.084572:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.084580:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.084582:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27832320, 27836416) 0 0 00000080:00200000:1.0:1789760778.084586:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.084600:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6795, 6795] 00020000:00200000:1.0:1789760778.084608:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27832320 chunk: [27832320, 27836416) 27836416, 4096 00020000:00200000:1.0:1789760778.084612:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.084627:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6795, 6795] 00000008:00010000:1.0:1789760778.084641:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.084646:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760778.084655:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27832320, 27836416) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.084666:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27832320=0x1a8b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.084711:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.084713:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27836416 00000020:00010000:1.0:1789760778.084720:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.084723:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.084726:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.084729:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.084732:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.084738:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.084743:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.084757:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760778.084851:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94700 - object 0x240000400:5 offset 27832320<>27836416 00000100:00100000:0.0:1789760778.084936:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512733056:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760778.113775:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974114 gen 3 00000008:00010000:0.0:1789760778.113818:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760778.113832:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.113840:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.113859:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760778.113880:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512733056:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760778.114060:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6795, 6795] 00000080:00200000:1.0:1789760778.114072:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.114077:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.114087:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27836416 i_blocks 55303 00000020:00200000:1.0:1789760778.114090:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.114098:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.114121:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27836416, count: 4096 00000080:00200000:1.0:1789760778.114127:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27836416, bytes: 4096 00000080:00200000:1.0:1789760778.114134:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.114141:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.114142:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27836416, 27840512) 0 0 00000080:00200000:1.0:1789760778.114146:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.114164:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6796, 6796] 00020000:00200000:1.0:1789760778.114171:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27836416 chunk: [27836416, 27840512) 27840512, 4096 00020000:00200000:1.0:1789760778.114175:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.114191:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6796, 6796] 00000008:00010000:1.0:1789760778.114203:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.114207:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760778.114216:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27836416, 27840512) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.114225:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27836416=0x1a8c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.114263:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.114265:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27840512 00000020:00010000:1.0:1789760778.114270:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.114273:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.114277:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.114279:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.114282:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.114287:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.114292:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.114303:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760778.114368:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 27836416<>27840512 00000100:00100000:0.0:1789760778.115000:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512733184:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760778.154754:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974115 gen 3 00000008:00010000:0.0:1789760778.154791:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760778.154796:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.154804:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.154832:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760778.154855:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512733184:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760778.154977:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6796, 6796] 00000080:00200000:1.0:1789760778.154992:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.154997:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.155009:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27840512 i_blocks 55303 00000020:00200000:1.0:1789760778.155012:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.155020:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.155047:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27840512, count: 4096 00000080:00200000:1.0:1789760778.155054:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27840512, bytes: 4096 00000080:00200000:1.0:1789760778.155061:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.155068:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.155070:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27840512, 27844608) 0 0 00000080:00200000:1.0:1789760778.155073:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.155085:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6797, 6797] 00020000:00200000:1.0:1789760778.155092:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27840512 chunk: [27840512, 27844608) 27844608, 4096 00020000:00200000:1.0:1789760778.155095:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.155111:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6797, 6797] 00000008:00010000:1.0:1789760778.155124:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.155129:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760778.155138:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27840512, 27844608) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.155158:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27840512=0x1a8d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.155202:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.155204:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27844608 00000020:00010000:1.0:1789760778.155211:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.155213:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.155216:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.155219:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.155221:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.155228:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.155233:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.155247:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760778.155319:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94380 - object 0x240000400:5 offset 27840512<>27844608 00000100:00100000:0.0:1789760778.157345:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760778.157374:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512733312:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760778.201084:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760778.201096:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.201107:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.201130:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760778.201161:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512733312:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760778.201251:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6797, 6797] 00000080:00200000:1.0:1789760778.201265:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.201270:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.201282:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27844608 i_blocks 55303 00000020:00200000:1.0:1789760778.201285:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.201294:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.201321:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27844608, count: 4096 00000080:00200000:1.0:1789760778.201328:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27844608, bytes: 4096 00000080:00200000:1.0:1789760778.201336:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.201343:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.201345:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27844608, 27848704) 0 0 00000080:00200000:1.0:1789760778.201349:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.201364:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6798, 6798] 00020000:00200000:1.0:1789760778.201371:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27844608 chunk: [27844608, 27848704) 27848704, 4096 00020000:00200000:1.0:1789760778.201375:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.201389:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6798, 6798] 00000008:00010000:1.0:1789760778.201403:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.201408:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760778.201427:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27844608, 27848704) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.201440:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27844608=0x1a8e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.201489:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.201492:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27848704 00000020:00010000:1.0:1789760778.201500:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.201503:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.201506:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.201510:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.201512:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.201518:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.201523:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.201548:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760778.201622:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 27844608<>27848704 00000100:00100000:0.0:1789760778.201862:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512733440:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760778.244870:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974116 gen 3 00000100:00100000:0.0:1789760778.244924:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005c96a00 x1876699512733440/t4294974117(4294974117) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760794 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760778.244970:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760778.244976:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.244983:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.245002:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760778.245020:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512733440:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760778.245287:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6798, 6798] 00000080:00200000:1.0:1789760778.245299:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.245304:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.245314:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27848704 i_blocks 55303 00000020:00200000:1.0:1789760778.245318:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.245325:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.245350:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27848704, count: 4096 00000080:00200000:1.0:1789760778.245357:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27848704, bytes: 4096 00000080:00200000:1.0:1789760778.245364:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.245370:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.245372:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27848704, 27852800) 0 0 00000080:00200000:1.0:1789760778.245376:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.245389:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6799, 6799] 00020000:00200000:1.0:1789760778.245396:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27848704 chunk: [27848704, 27852800) 27852800, 4096 00020000:00200000:1.0:1789760778.245400:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.245414:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6799, 6799] 00000008:00010000:1.0:1789760778.245428:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.245433:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760778.245442:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27848704, 27852800) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.245452:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27848704=0x1a8f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.245494:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.245497:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27852800 00000020:00010000:1.0:1789760778.245503:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.245506:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.245509:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.245512:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.245514:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.245520:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.245525:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.245550:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760778.245624:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 27848704<>27852800 00000100:00100000:0.0:1789760778.245753:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760778.245766:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512733568:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760778.280243:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974118 gen 3 00000008:00010000:0.0:1789760778.280289:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760778.280295:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.280303:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.280324:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760778.280346:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512733568:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760778.280930:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6799, 6799] 00000080:00200000:1.0:1789760778.281332:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.281338:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.281351:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27852800 i_blocks 55303 00000020:00200000:1.0:1789760778.281354:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.281364:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.281390:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27852800, count: 4096 00000080:00200000:1.0:1789760778.281397:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27852800, bytes: 4096 00000080:00200000:1.0:1789760778.281406:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.281413:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.281415:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27852800, 27856896) 0 0 00000080:00200000:1.0:1789760778.281424:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.281438:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6800, 6800] 00020000:00200000:1.0:1789760778.281444:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27852800 chunk: [27852800, 27856896) 27856896, 4096 00020000:00200000:1.0:1789760778.281448:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.281464:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6800, 6800] 00000008:00010000:1.0:1789760778.281478:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.281482:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760778.281491:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27852800, 27856896) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.281503:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27852800=0x1a90000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.281669:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.281672:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27856896 00000020:00010000:1.0:1789760778.281679:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.281683:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.281686:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.281689:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.281692:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.281699:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.281705:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.281723:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760778.281800:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 27852800<>27856896 00000100:00100000:0.0:1789760778.282019:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512733696:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760778.344071:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974119 gen 3 00000008:00010000:0.0:1789760778.344106:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760778.344112:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.344121:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.344142:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760778.344176:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512733696:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760778.344193:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6800, 6800] 00000080:00200000:1.0:1789760778.344207:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.344211:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.344222:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27856896 i_blocks 55303 00000020:00200000:1.0:1789760778.344225:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.344232:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.344261:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27856896, count: 4096 00000080:00200000:1.0:1789760778.344268:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27856896, bytes: 4096 00000080:00200000:1.0:1789760778.344275:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.344281:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.344283:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27856896, 27860992) 0 0 00000080:00200000:1.0:1789760778.344286:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.344300:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6801, 6801] 00020000:00200000:1.0:1789760778.344307:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27856896 chunk: [27856896, 27860992) 27860992, 4096 00020000:00200000:1.0:1789760778.344311:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.344325:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6801, 6801] 00000008:00010000:1.0:1789760778.344337:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.344341:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760778.344349:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27856896, 27860992) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.344358:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27856896=0x1a91000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.344401:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.344404:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27860992 00000020:00010000:1.0:1789760778.344411:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.344414:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.344417:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.344419:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.344422:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.344428:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.344433:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.344446:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760778.344516:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97480 - object 0x240000400:5 offset 27856896<>27860992 00000100:00100000:0.0:1789760778.345126:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512733824:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760778.376749:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974120 gen 3 00000008:00010000:0.0:1789760778.376784:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760778.376931:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.376940:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.376960:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760778.376984:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512733824:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760778.377034:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6801, 6801] 00000080:00200000:1.0:1789760778.377046:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.377051:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.377063:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27860992 i_blocks 55303 00000020:00200000:1.0:1789760778.377065:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.377072:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.377092:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27860992, count: 4096 00000080:00200000:1.0:1789760778.377097:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27860992, bytes: 4096 00000080:00200000:1.0:1789760778.377103:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.377109:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.377110:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27860992, 27865088) 0 0 00000080:00200000:1.0:1789760778.377113:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.377122:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6802, 6802] 00020000:00200000:1.0:1789760778.377128:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27860992 chunk: [27860992, 27865088) 27865088, 4096 00020000:00200000:1.0:1789760778.377130:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.377141:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6802, 6802] 00000008:00010000:1.0:1789760778.377162:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.377225:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760778.377238:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27860992, 27865088) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.377248:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27860992=0x1a92000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.377293:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.377296:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27865088 00000020:00010000:1.0:1789760778.377302:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.377305:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.377307:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.377310:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.377313:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.377319:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.377324:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.377338:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760778.377438:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96680 - object 0x240000400:5 offset 27860992<>27865088 00000100:00100000:0.0:1789760778.377481:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760778.377494:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512733952:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760778.429805:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974121 gen 3 00000008:00010000:0.0:1789760778.429842:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760778.429848:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.429857:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.429984:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760778.430009:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512733952:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760778.432781:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6802, 6802] 00000080:00200000:1.0:1789760778.432801:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.432807:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.432820:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27865088 i_blocks 55303 00000020:00200000:1.0:1789760778.432824:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.432837:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.432862:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27865088, count: 4096 00000080:00200000:1.0:1789760778.432869:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27865088, bytes: 4096 00000080:00200000:1.0:1789760778.432881:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.432888:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.432890:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27865088, 27869184) 0 0 00000080:00200000:1.0:1789760778.432894:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.432925:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6803, 6803] 00020000:00200000:1.0:1789760778.432932:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27865088 chunk: [27865088, 27869184) 27869184, 4096 00020000:00200000:1.0:1789760778.432936:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.432952:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6803, 6803] 00000008:00010000:1.0:1789760778.432967:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.432971:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760778.432980:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27865088, 27869184) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.432992:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27865088=0x1a93000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.433041:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.433044:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27869184 00000020:00010000:1.0:1789760778.433051:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.433055:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.433058:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.433060:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.433063:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.433069:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.433076:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.433100:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760778.433206:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95880 - object 0x240000400:5 offset 27865088<>27869184 00000100:00100000:0.0:1789760778.433370:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760778.433390:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512734080:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760778.470572:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974122 gen 3 00000008:00010000:0.0:1789760778.470606:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760778.470612:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.470621:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.470636:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760778.470658:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512734080:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760778.470681:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6803, 6803] 00000080:00200000:1.0:1789760778.470695:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.470769:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.470782:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27869184 i_blocks 55303 00000020:00200000:1.0:1789760778.470785:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.470793:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.470828:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27869184, count: 4096 00000080:00200000:1.0:1789760778.470836:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27869184, bytes: 4096 00000080:00200000:1.0:1789760778.470843:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.470851:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.470854:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27869184, 27873280) 0 0 00000080:00200000:1.0:1789760778.470857:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.470870:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6804, 6804] 00020000:00200000:1.0:1789760778.470875:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27869184 chunk: [27869184, 27873280) 27873280, 4096 00020000:00200000:1.0:1789760778.470879:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.470892:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6804, 6804] 00000008:00010000:1.0:1789760778.470930:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.470935:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760778.471054:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27869184, 27873280) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.471067:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27869184=0x1a94000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.471109:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.471112:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27873280 00000020:00010000:1.0:1789760778.471130:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.471134:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.471137:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.471140:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.471142:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.471159:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.471165:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.471180:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760778.471247:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96d80 - object 0x240000400:5 offset 27869184<>27873280 00000100:00100000:0.0:1789760778.471409:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760778.471425:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512734208:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760778.507629:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974123 gen 3 00000008:00010000:0.0:1789760778.507670:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760778.507676:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.507684:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.507708:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760778.507732:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512734208:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760778.507924:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6804, 6804] 00000080:00200000:1.0:1789760778.507937:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.507943:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.507955:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27873280 i_blocks 55303 00000020:00200000:1.0:1789760778.507958:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.507967:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.508304:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27873280, count: 4096 00000080:00200000:1.0:1789760778.508311:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27873280, bytes: 4096 00000080:00200000:1.0:1789760778.508320:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.508328:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.508330:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27873280, 27877376) 0 0 00000080:00200000:1.0:1789760778.508334:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.508347:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6805, 6805] 00020000:00200000:1.0:1789760778.508354:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27873280 chunk: [27873280, 27877376) 27877376, 4096 00020000:00200000:1.0:1789760778.508358:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.508372:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6805, 6805] 00000008:00010000:1.0:1789760778.508386:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.508391:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760778.508400:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27873280, 27877376) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.508410:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27873280=0x1a95000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.508454:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.508457:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27877376 00000020:00010000:1.0:1789760778.508463:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.508467:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.508469:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.508472:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.508475:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.508482:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.508488:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.508502:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760778.508693:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96d80 - object 0x240000400:5 offset 27873280<>27877376 00000100:00100000:1.0:1789760778.509083:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512734336:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760778.551315:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760778.551326:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760778.551334:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760778.551356:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760778.551381:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512734336:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760778.551545:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6805, 6805] 00000080:00200000:2.0:1789760778.551558:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760778.551563:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.551573:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27877376 i_blocks 55303 00000020:00200000:2.0:1789760778.551576:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.551584:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.551608:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27877376, count: 4096 00000080:00200000:2.0:1789760778.551614:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27877376, bytes: 4096 00000080:00200000:2.0:1789760778.551621:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.551627:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.551629:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27877376, 27881472) 0 0 00000080:00200000:2.0:1789760778.551633:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.551646:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6806, 6806] 00020000:00200000:2.0:1789760778.551652:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27877376 chunk: [27877376, 27881472) 27881472, 4096 00020000:00200000:2.0:1789760778.551655:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.551670:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6806, 6806] 00000008:00010000:2.0:1789760778.551683:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.551687:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760778.551695:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27877376, 27881472) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.551705:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27877376=0x1a96000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.551745:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.551748:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27881472 00000020:00010000:2.0:1789760778.551754:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.551757:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.551759:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.551762:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.551765:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.551770:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.551776:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760778.551788:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760778.551875:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 27877376<>27881472 00000100:00100000:3.0:1789760778.551933:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512734464:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760778.591659:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974125 gen 3 00000008:00010000:2.0:1789760778.591707:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:2.0:1789760778.591714:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.591722:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.591834:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:2.0:1789760778.591858:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512734464:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760778.592044:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6806, 6806] 00000080:00200000:1.0:1789760778.592057:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.592062:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.592074:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27881472 i_blocks 55303 00000020:00200000:1.0:1789760778.592077:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.592085:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.592110:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27881472, count: 4096 00000080:00200000:1.0:1789760778.592116:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27881472, bytes: 4096 00000080:00200000:1.0:1789760778.592124:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.592130:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.592132:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27881472, 27885568) 0 0 00000080:00200000:1.0:1789760778.592135:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.592162:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6807, 6807] 00020000:00200000:1.0:1789760778.592169:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27881472 chunk: [27881472, 27885568) 27885568, 4096 00020000:00200000:1.0:1789760778.592173:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.592189:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6807, 6807] 00000008:00010000:1.0:1789760778.592202:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.592207:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760778.592216:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27881472, 27885568) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.592226:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27881472=0x1a97000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.592269:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.592271:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27885568 00000020:00010000:1.0:1789760778.592278:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.592281:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.592284:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.592287:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.592290:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.592296:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.592301:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.592315:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760778.592389:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95880 - object 0x240000400:5 offset 27881472<>27885568 00000100:00100000:0.0:1789760778.592461:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512734592:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760778.628841:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760778.628857:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760778.628865:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760778.629101:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760778.629127:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512734592:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760778.629364:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6807, 6807] 00000080:00200000:2.0:1789760778.629376:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760778.629382:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.629394:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27885568 i_blocks 55303 00000020:00200000:2.0:1789760778.629397:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.629405:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.629431:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27885568, count: 4096 00000080:00200000:2.0:1789760778.629437:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27885568, bytes: 4096 00000080:00200000:2.0:1789760778.629445:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.629452:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.629454:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27885568, 27889664) 0 0 00000080:00200000:2.0:1789760778.629457:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.629470:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6808, 6808] 00020000:00200000:2.0:1789760778.629477:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27885568 chunk: [27885568, 27889664) 27889664, 4096 00020000:00200000:2.0:1789760778.629481:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.629495:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6808, 6808] 00000008:00010000:2.0:1789760778.629510:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.629514:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760778.629523:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27885568, 27889664) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.629532:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27885568=0x1a98000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.629677:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.629680:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27889664 00000020:00010000:2.0:1789760778.629687:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.629690:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.629693:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.629696:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.629699:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.629705:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.629710:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760778.629723:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760778.629798:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 27885568<>27889664 00000100:00100000:3.0:1789760778.630283:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512734720:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760778.663473:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974127 gen 3 00000008:00010000:2.0:1789760778.663521:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:2.0:1789760778.663529:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.663537:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.663560:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:2.0:1789760778.663585:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512734720:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760778.663733:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6808, 6808] 00000080:00200000:0.0:1789760778.663745:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760778.663750:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760778.663762:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27889664 i_blocks 55303 00000020:00200000:0.0:1789760778.663765:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760778.663774:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760778.663799:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27889664, count: 4096 00000080:00200000:0.0:1789760778.663806:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27889664, bytes: 4096 00000080:00200000:0.0:1789760778.663815:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760778.663822:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760778.663824:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27889664, 27893760) 0 0 00000080:00200000:0.0:1789760778.663828:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760778.663842:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6809, 6809] 00020000:00200000:0.0:1789760778.663848:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27889664 chunk: [27889664, 27893760) 27893760, 4096 00020000:00200000:0.0:1789760778.663852:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760778.663866:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6809, 6809] 00000008:00010000:0.0:1789760778.663879:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760778.663883:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760778.663894:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27889664, 27893760) append 0 append lockpos 0 00000080:00200000:0.0:1789760778.663923:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27889664=0x1a99000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760778.663978:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760778.663980:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27893760 00000020:00010000:0.0:1789760778.663987:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760778.663991:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.663994:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.663997:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760778.663999:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760778.664005:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760778.664010:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760778.664024:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc180 00000008:00100000:0.0:1789760778.664110:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148700 - object 0x240000400:5 offset 27889664<>27893760 00000100:00100000:0.0:1789760778.664161:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760778.664176:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512734848:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760778.702098:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974128 gen 3 00000008:00010000:0.0:1789760778.702133:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:0.0:1789760778.702141:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.702161:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.702184:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:0.0:1789760778.702207:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512734848:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760778.702356:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6809, 6809] 00000080:00200000:2.0:1789760778.702370:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760778.702374:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.702386:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27893760 i_blocks 55303 00000020:00200000:2.0:1789760778.702389:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.702399:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.702427:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27893760, count: 4096 00000080:00200000:2.0:1789760778.702433:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27893760, bytes: 4096 00000080:00200000:2.0:1789760778.702441:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.702449:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.702451:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27893760, 27897856) 0 0 00000080:00200000:2.0:1789760778.702455:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.702468:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6810, 6810] 00020000:00200000:2.0:1789760778.702476:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27893760 chunk: [27893760, 27897856) 27897856, 4096 00020000:00200000:2.0:1789760778.702480:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.702496:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6810, 6810] 00000008:00010000:2.0:1789760778.702510:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.702515:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760778.702525:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27893760, 27897856) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.702597:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27893760=0x1a9a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.702642:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.702644:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27897856 00000020:00010000:2.0:1789760778.702652:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.702655:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.702658:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.702661:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.702663:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.702669:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.702675:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760778.702689:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760778.702774:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 27893760<>27897856 00000100:00100000:3.0:1789760778.702987:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512734976:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760778.747495:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:2.0:1789760778.747506:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.747514:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.747597:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:2.0:1789760778.747623:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512734976:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760778.748023:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6810, 6810] 00000080:00200000:0.0:1789760778.748036:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760778.748042:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760778.748054:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27897856 i_blocks 55303 00000020:00200000:0.0:1789760778.748058:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760778.748066:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760778.748090:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27897856, count: 4096 00000080:00200000:0.0:1789760778.748097:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27897856, bytes: 4096 00000080:00200000:0.0:1789760778.748105:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760778.748113:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760778.748115:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27897856, 27901952) 0 0 00000080:00200000:0.0:1789760778.748119:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760778.748131:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6811, 6811] 00020000:00200000:0.0:1789760778.748138:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27897856 chunk: [27897856, 27901952) 27901952, 4096 00020000:00200000:0.0:1789760778.748142:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760778.748168:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6811, 6811] 00000008:00010000:0.0:1789760778.748181:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760778.748185:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760778.748193:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27897856, 27901952) append 0 append lockpos 0 00000080:00200000:0.0:1789760778.748203:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27897856=0x1a9b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760778.748245:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760778.748248:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27901952 00000020:00010000:0.0:1789760778.748254:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760778.748257:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.748260:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.748262:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760778.748265:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760778.748271:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760778.748276:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760778.748290:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc180 00000008:00100000:0.0:1789760778.748366:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148700 - object 0x240000400:5 offset 27897856<>27901952 00000100:00100000:1.0:1789760778.748442:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512735104:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760778.779326:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974130 gen 3 00000008:00010000:1.0:1789760778.779372:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760778.779380:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760778.779389:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760778.779411:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760778.779434:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512735104:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760778.779526:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6811, 6811] 00000080:00200000:0.0:1789760778.779545:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760778.779550:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760778.779560:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27901952 i_blocks 55303 00000020:00200000:0.0:1789760778.779564:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760778.779571:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760778.779596:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27901952, count: 4096 00000080:00200000:0.0:1789760778.779602:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27901952, bytes: 4096 00000080:00200000:0.0:1789760778.779611:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760778.779619:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760778.779621:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27901952, 27906048) 0 0 00000080:00200000:0.0:1789760778.779624:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760778.779637:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6812, 6812] 00020000:00200000:0.0:1789760778.779643:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27901952 chunk: [27901952, 27906048) 27906048, 4096 00020000:00200000:0.0:1789760778.779647:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760778.779660:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6812, 6812] 00000008:00010000:0.0:1789760778.779674:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760778.779679:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760778.779688:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27901952, 27906048) append 0 append lockpos 0 00000080:00200000:0.0:1789760778.779698:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27901952=0x1a9c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760778.779741:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760778.779745:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27906048 00000020:00010000:0.0:1789760778.779752:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760778.779755:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.779758:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.779760:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760778.779763:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760778.779769:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760778.779774:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760778.779787:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000008:00100000:0.0:1789760778.780069:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514ad80 - object 0x240000400:5 offset 27901952<>27906048 00000100:00100000:1.0:1789760778.780116:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760778.780461:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512735232:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760778.808215:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974131 gen 3 00000008:00010000:1.0:1789760778.808259:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760778.808265:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760778.808274:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760778.808347:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760778.808370:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512735232:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760778.809101:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6812, 6812] 00000080:00200000:2.0:1789760778.809116:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760778.809187:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.809202:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27906048 i_blocks 55303 00000020:00200000:2.0:1789760778.809207:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.809216:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.809241:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27906048, count: 4096 00000080:00200000:2.0:1789760778.809247:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27906048, bytes: 4096 00000080:00200000:2.0:1789760778.809256:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.809263:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.809265:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27906048, 27910144) 0 0 00000080:00200000:2.0:1789760778.809269:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.809282:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6813, 6813] 00020000:00200000:2.0:1789760778.809289:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27906048 chunk: [27906048, 27910144) 27910144, 4096 00020000:00200000:2.0:1789760778.809293:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.809308:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6813, 6813] 00000008:00010000:2.0:1789760778.809321:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.809326:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760778.809336:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27906048, 27910144) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.809345:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27906048=0x1a9d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.809393:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.809396:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27910144 00000020:00010000:2.0:1789760778.809403:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.809407:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.809409:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.809412:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.809414:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.809422:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.809427:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760778.809445:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760778.809523:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 27906048<>27910144 00000100:00100000:3.0:1789760778.809697:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512735360:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760778.843196:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:2.0:1789760778.843209:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.843220:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.843350:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:2.0:1789760778.843381:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512735360:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760778.843446:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6813, 6813] 00000080:00200000:1.0:1789760778.843458:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760778.843461:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760778.843472:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27910144 i_blocks 55303 00000020:00200000:1.0:1789760778.843474:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760778.843481:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760778.843511:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27910144, count: 4096 00000080:00200000:1.0:1789760778.843515:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27910144, bytes: 4096 00000080:00200000:1.0:1789760778.843523:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760778.843529:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760778.843531:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27910144, 27914240) 0 0 00000080:00200000:1.0:1789760778.843534:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760778.843550:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6814, 6814] 00020000:00200000:1.0:1789760778.843558:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27910144 chunk: [27910144, 27914240) 27914240, 4096 00020000:00200000:1.0:1789760778.843562:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760778.843577:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6814, 6814] 00000008:00010000:1.0:1789760778.843592:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760778.843596:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760778.843606:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27910144, 27914240) append 0 append lockpos 0 00000080:00200000:1.0:1789760778.843616:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27910144=0x1a9e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760778.843663:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760778.843666:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27914240 00000020:00010000:1.0:1789760778.843673:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760778.843676:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.843680:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760778.843683:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760778.843685:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760778.843691:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760778.843696:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760778.843708:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760778.843773:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96680 - object 0x240000400:5 offset 27910144<>27914240 00000100:00100000:0.0:1789760778.843886:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512735488:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760778.880315:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974133 gen 3 00000008:00010000:0.0:1789760778.880362:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760778.880368:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760778.880375:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760778.880454:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760778.880478:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512735488:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760778.881259:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6814, 6814] 00000080:00200000:2.0:1789760778.881274:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760778.881279:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760778.881291:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27914240 i_blocks 55303 00000020:00200000:2.0:1789760778.881294:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760778.881303:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760778.881328:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27914240, count: 4096 00000080:00200000:2.0:1789760778.881334:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27914240, bytes: 4096 00000080:00200000:2.0:1789760778.881344:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760778.881351:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760778.881353:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27914240, 27918336) 0 0 00000080:00200000:2.0:1789760778.881357:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760778.881370:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6815, 6815] 00020000:00200000:2.0:1789760778.881400:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27914240 chunk: [27914240, 27918336) 27918336, 4096 00020000:00200000:2.0:1789760778.881405:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760778.881420:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6815, 6815] 00000008:00010000:2.0:1789760778.881434:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760778.881439:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760778.881447:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27914240, 27918336) append 0 append lockpos 0 00000080:00200000:2.0:1789760778.881458:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27914240=0x1a9f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760778.881504:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760778.881507:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27918336 00000020:00010000:2.0:1789760778.881513:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760778.881516:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.881519:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760778.881522:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760778.881525:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760778.881531:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760778.881537:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760778.881554:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760778.881632:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 27914240<>27918336 00000100:00100000:2.0:1789760778.881682:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760778.881700:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512735616:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760778.936928:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:2.0:1789760778.936939:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760778.936947:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760778.936966:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:2.0:1789760778.936990:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512735616:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760778.937012:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6815, 6815] 00000080:00200000:0.0:1789760778.937026:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760778.937031:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760778.937045:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27918336 i_blocks 55303 00000020:00200000:0.0:1789760778.937049:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760778.937057:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760778.937083:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27918336, count: 4096 00000080:00200000:0.0:1789760778.937089:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27918336, bytes: 4096 00000080:00200000:0.0:1789760778.937097:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760778.937104:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760778.937106:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27918336, 27922432) 0 0 00000080:00200000:0.0:1789760778.937110:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760778.937123:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6816, 6816] 00020000:00200000:0.0:1789760778.937129:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27918336 chunk: [27918336, 27922432) 27922432, 4096 00020000:00200000:0.0:1789760778.937132:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760778.937160:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6816, 6816] 00000008:00010000:0.0:1789760778.937175:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760778.937180:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760778.937190:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27918336, 27922432) append 0 append lockpos 0 00000080:00200000:0.0:1789760778.937199:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27918336=0x1aa0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760778.937240:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760778.937243:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27922432 00000020:00010000:0.0:1789760778.937250:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760778.937254:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.937257:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.937261:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760778.937265:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760778.937272:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760778.937278:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760778.937293:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccc00 00000008:00100000:0.0:1789760778.937367:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148a80 - object 0x240000400:5 offset 27918336<>27922432 00000100:00100000:1.0:1789760778.937488:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512735744:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760778.992590:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974135 gen 3 00000008:00010000:1.0:1789760778.992631:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760778.992637:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760778.992645:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760778.992666:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760778.992686:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512735744:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760778.992824:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6816, 6816] 00000080:00200000:0.0:1789760778.992835:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760778.992839:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760778.992850:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27922432 i_blocks 55303 00000020:00200000:0.0:1789760778.992853:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760778.992860:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760778.992883:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27922432, count: 4096 00000080:00200000:0.0:1789760778.992889:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27922432, bytes: 4096 00000080:00200000:0.0:1789760778.992896:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760778.993081:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760778.993084:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27922432, 27926528) 0 0 00000080:00200000:0.0:1789760778.993088:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760778.993101:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6817, 6817] 00020000:00200000:0.0:1789760778.993107:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27922432 chunk: [27922432, 27926528) 27926528, 4096 00020000:00200000:0.0:1789760778.993111:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760778.993135:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6817, 6817] 00000008:00010000:0.0:1789760778.993156:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760778.993161:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760778.993171:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27922432, 27926528) append 0 append lockpos 0 00000080:00200000:0.0:1789760778.993181:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27922432=0x1aa1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760778.993645:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760778.993648:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27926528 00000020:00010000:0.0:1789760778.993655:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760778.993658:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.993661:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760778.993664:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760778.993667:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760778.993673:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760778.993678:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760778.993697:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cce40 00000008:00100000:0.0:1789760778.993768:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148e00 - object 0x240000400:5 offset 27922432<>27926528 00000100:00100000:1.0:1789760778.993986:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760778.994002:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512735872:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760779.037520:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760779.044301:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974136 gen 3 00000008:00010000:1.0:1789760779.044334:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760779.044339:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.044348:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.044369:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760779.044391:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512735872:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760779.045078:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6817, 6817] 00000080:00200000:0.0:1789760779.045093:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.045099:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.045112:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27926528 i_blocks 55303 00000020:00200000:0.0:1789760779.045116:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.045126:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.045164:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27926528, count: 4096 00000080:00200000:0.0:1789760779.045172:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27926528, bytes: 4096 00000080:00200000:0.0:1789760779.045182:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.045190:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.045192:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27926528, 27930624) 0 0 00000080:00200000:0.0:1789760779.045197:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.045214:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6818, 6818] 00020000:00200000:0.0:1789760779.045222:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27926528 chunk: [27926528, 27930624) 27930624, 4096 00020000:00200000:0.0:1789760779.045227:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.045242:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6818, 6818] 00000008:00010000:0.0:1789760779.045258:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.045263:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760779.045273:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27926528, 27930624) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.045287:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27926528=0x1aa2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.045340:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.045343:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27930624 00000020:00010000:0.0:1789760779.045351:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.045354:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.045357:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.045360:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.045362:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.045369:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.045375:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.045389:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cce40 00000008:00100000:0.0:1789760779.045464:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a680 - object 0x240000400:5 offset 27926528<>27930624 00000100:00100000:1.0:1789760779.046251:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512736000:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760779.102893:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974137 gen 3 00000008:00010000:1.0:1789760779.102947:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760779.102954:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.102963:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.103043:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760779.103065:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512736000:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760779.107789:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6818, 6818] 00000080:00200000:0.0:1789760779.107808:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.107814:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.107829:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27930624 i_blocks 55303 00000020:00200000:0.0:1789760779.107833:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.107845:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.107870:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27930624, count: 4096 00000080:00200000:0.0:1789760779.107876:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27930624, bytes: 4096 00000080:00200000:0.0:1789760779.107887:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.107894:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.107897:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27930624, 27934720) 0 0 00000080:00200000:0.0:1789760779.107916:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.107931:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6819, 6819] 00020000:00200000:0.0:1789760779.107938:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27930624 chunk: [27930624, 27934720) 27934720, 4096 00020000:00200000:0.0:1789760779.107942:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.107957:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6819, 6819] 00000008:00010000:0.0:1789760779.107971:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.107975:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760779.107984:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27930624, 27934720) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.107995:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27930624=0x1aa3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.108046:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.108049:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27934720 00000020:00010000:0.0:1789760779.108054:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.108057:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.108059:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.108062:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.108064:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.108070:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.108076:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.108098:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccc00 00000008:00100000:0.0:1789760779.108193:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a300 - object 0x240000400:5 offset 27930624<>27934720 00000100:00100000:1.0:1789760779.108324:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512736128:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760779.160511:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974138 gen 3 00000008:00010000:1.0:1789760779.160549:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760779.160555:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.160565:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.160584:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760779.160607:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512736128:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760779.160895:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6819, 6819] 00000080:00200000:0.0:1789760779.160939:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.160945:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.160956:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27934720 i_blocks 55303 00000020:00200000:0.0:1789760779.160960:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.160967:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.160993:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27934720, count: 4096 00000080:00200000:0.0:1789760779.160999:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27934720, bytes: 4096 00000080:00200000:0.0:1789760779.161007:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.161014:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.161016:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27934720, 27938816) 0 0 00000080:00200000:0.0:1789760779.161020:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.161033:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6820, 6820] 00020000:00200000:0.0:1789760779.161039:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27934720 chunk: [27934720, 27938816) 27938816, 4096 00020000:00200000:0.0:1789760779.161043:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.161058:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6820, 6820] 00000008:00010000:0.0:1789760779.161071:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.161076:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760779.161085:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27934720, 27938816) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.161096:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27934720=0x1aa4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.161253:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.161256:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27938816 00000020:00010000:0.0:1789760779.161264:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.161267:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.161270:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.161273:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.161276:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.161282:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.161288:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.161302:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000008:00100000:0.0:1789760779.161377:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149880 - object 0x240000400:5 offset 27934720<>27938816 00000100:00100000:1.0:1789760779.161464:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512736256:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760779.190841:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974139 gen 3 00000008:00010000:1.0:1789760779.190878:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760779.190885:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.190894:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.190947:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760779.190971:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512736256:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760779.191229:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6820, 6820] 00000080:00200000:0.0:1789760779.191241:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.191247:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.191258:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27938816 i_blocks 55303 00000020:00200000:0.0:1789760779.191261:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.191270:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.191292:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27938816, count: 4096 00000080:00200000:0.0:1789760779.191297:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27938816, bytes: 4096 00000080:00200000:0.0:1789760779.191304:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.191311:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.191313:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27938816, 27942912) 0 0 00000080:00200000:0.0:1789760779.191317:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.191330:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6821, 6821] 00020000:00200000:0.0:1789760779.191336:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27938816 chunk: [27938816, 27942912) 27942912, 4096 00020000:00200000:0.0:1789760779.191340:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.191355:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6821, 6821] 00000008:00010000:0.0:1789760779.191368:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.191373:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760779.191382:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27938816, 27942912) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.191392:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27938816=0x1aa5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.191432:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.191435:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27942912 00000020:00010000:0.0:1789760779.191441:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.191446:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.191448:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.191451:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.191453:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.191459:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.191466:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.191479:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc180 00000008:00100000:0.0:1789760779.191622:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148380 - object 0x240000400:5 offset 27938816<>27942912 00000100:00100000:1.0:1789760779.191719:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512736384:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760779.234354:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974140 gen 3 00000008:00010000:1.0:1789760779.234392:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760779.234398:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.234407:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.234486:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760779.234505:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512736384:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760779.234567:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6821, 6821] 00000080:00200000:0.0:1789760779.234580:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.234585:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.234597:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27942912 i_blocks 55303 00000020:00200000:0.0:1789760779.234600:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.234607:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.234631:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27942912, count: 4096 00000080:00200000:0.0:1789760779.234638:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27942912, bytes: 4096 00000080:00200000:0.0:1789760779.234657:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.234664:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.234666:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27942912, 27947008) 0 0 00000080:00200000:0.0:1789760779.234670:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.234684:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6822, 6822] 00020000:00200000:0.0:1789760779.234691:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27942912 chunk: [27942912, 27947008) 27947008, 4096 00020000:00200000:0.0:1789760779.234695:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.234709:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6822, 6822] 00000008:00010000:0.0:1789760779.234723:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.234727:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760779.234737:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27942912, 27947008) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.234747:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27942912=0x1aa6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.234974:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.234978:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27947008 00000020:00010000:0.0:1789760779.234985:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.234987:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.234989:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.234992:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.234993:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.234998:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.235002:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.235015:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccd80 00000008:00100000:0.0:1789760779.235137:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b800 - object 0x240000400:5 offset 27942912<>27947008 00000100:00100000:0.0:1789760779.235180:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760779.235193:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512736512:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760779.267067:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974141 gen 3 00000008:00010000:0.0:1789760779.267104:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:0.0:1789760779.267111:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760779.267120:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760779.267151:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:0.0:1789760779.267174:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512736512:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760779.267950:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6822, 6822] 00000080:00200000:1.0:1789760779.267967:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760779.267972:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760779.267984:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27947008 i_blocks 55303 00000020:00200000:1.0:1789760779.267988:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760779.267997:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760779.268023:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27947008, count: 4096 00000080:00200000:1.0:1789760779.268029:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27947008, bytes: 4096 00000080:00200000:1.0:1789760779.268037:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760779.268044:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760779.268047:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27947008, 27951104) 0 0 00000080:00200000:1.0:1789760779.268050:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760779.268063:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6823, 6823] 00020000:00200000:1.0:1789760779.268069:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27947008 chunk: [27947008, 27951104) 27951104, 4096 00020000:00200000:1.0:1789760779.268073:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760779.268087:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6823, 6823] 00000008:00010000:1.0:1789760779.268100:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760779.268106:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760779.268116:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27947008, 27951104) append 0 append lockpos 0 00000080:00200000:1.0:1789760779.268126:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27947008=0x1aa7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760779.268256:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760779.268259:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27951104 00000020:00010000:1.0:1789760779.268267:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760779.268270:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.268273:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760779.268276:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760779.268279:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760779.268286:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760779.268292:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760779.268309:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760779.268383:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97480 - object 0x240000400:5 offset 27947008<>27951104 00000100:00100000:1.0:1789760779.268426:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760779.268443:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512736640:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760779.304334:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974142 gen 3 00000008:00010000:1.0:1789760779.304370:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760779.304377:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.304386:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.304407:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760779.304428:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512736640:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760779.304521:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6823, 6823] 00000080:00200000:0.0:1789760779.304534:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.304547:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.304559:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27951104 i_blocks 55303 00000020:00200000:0.0:1789760779.304562:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.304570:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.304598:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27951104, count: 4096 00000080:00200000:0.0:1789760779.304604:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27951104, bytes: 4096 00000080:00200000:0.0:1789760779.304612:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.304619:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.304622:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27951104, 27955200) 0 0 00000080:00200000:0.0:1789760779.304625:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.304638:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6824, 6824] 00020000:00200000:0.0:1789760779.304645:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27951104 chunk: [27951104, 27955200) 27955200, 4096 00020000:00200000:0.0:1789760779.304648:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.304663:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6824, 6824] 00000008:00010000:0.0:1789760779.304677:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.304682:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760779.304693:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27951104, 27955200) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.304702:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27951104=0x1aa8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.304747:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.304749:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27955200 00000020:00010000:0.0:1789760779.304755:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.304758:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.304762:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.304765:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.304768:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.304774:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.304780:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.304805:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccd80 00000008:00100000:0.0:1789760779.304885:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b800 - object 0x240000400:5 offset 27951104<>27955200 00000100:00100000:1.0:1789760779.305254:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512736768:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760779.355615:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974143 gen 3 00000008:00010000:1.0:1789760779.355651:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760779.355658:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.355666:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.355798:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760779.355822:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512736768:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760779.356102:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6824, 6824] 00000080:00200000:2.0:1789760779.356115:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.356120:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.356131:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27955200 i_blocks 55303 00000020:00200000:2.0:1789760779.356134:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.356153:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.356178:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27955200, count: 4096 00000080:00200000:2.0:1789760779.356184:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27955200, bytes: 4096 00000080:00200000:2.0:1789760779.356191:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.356199:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.356201:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27955200, 27959296) 0 0 00000080:00200000:2.0:1789760779.356205:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.356217:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6825, 6825] 00020000:00200000:2.0:1789760779.356223:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27955200 chunk: [27955200, 27959296) 27959296, 4096 00020000:00200000:2.0:1789760779.356227:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.356240:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6825, 6825] 00000008:00010000:2.0:1789760779.356253:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.356257:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760779.356266:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27955200, 27959296) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.356277:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27955200=0x1aa9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.356318:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.356322:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27959296 00000020:00010000:2.0:1789760779.356328:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.356331:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.356334:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.356337:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.356340:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.356346:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.356351:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760779.356442:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760779.356515:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 27955200<>27959296 00000100:00100000:3.0:1789760779.356644:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512736896:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760779.403862:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974144 gen 3 00000008:00010000:2.0:1789760779.403894:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:2.0:1789760779.403928:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760779.403937:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760779.403961:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:2.0:1789760779.403983:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512736896:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760779.404079:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6825, 6825] 00000080:00200000:0.0:1789760779.404091:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.404097:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.404109:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27959296 i_blocks 55303 00000020:00200000:0.0:1789760779.404113:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.404120:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.404155:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27959296, count: 4096 00000080:00200000:0.0:1789760779.404162:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27959296, bytes: 4096 00000080:00200000:0.0:1789760779.404169:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.404176:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.404178:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27959296, 27963392) 0 0 00000080:00200000:0.0:1789760779.404182:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.404194:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6826, 6826] 00020000:00200000:0.0:1789760779.404200:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27959296 chunk: [27959296, 27963392) 27963392, 4096 00020000:00200000:0.0:1789760779.404204:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.404228:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6826, 6826] 00000008:00010000:0.0:1789760779.404241:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.404245:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760779.404255:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27959296, 27963392) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.404264:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27959296=0x1aaa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.404305:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.404308:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27963392 00000020:00010000:0.0:1789760779.404314:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.404317:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.404320:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.404323:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.404326:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.404332:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.404337:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.404349:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc780 00000008:00100000:0.0:1789760779.404436:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b480 - object 0x240000400:5 offset 27959296<>27963392 00000100:00100000:1.0:1789760779.404532:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512737024:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760779.471041:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974145 gen 3 00000008:00010000:1.0:1789760779.471077:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760779.471084:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.471093:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.471114:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760779.471137:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512737024:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760779.471243:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6826, 6826] 00000080:00200000:0.0:1789760779.471257:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.471263:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.471274:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27963392 i_blocks 55303 00000020:00200000:0.0:1789760779.471278:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.471285:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.471310:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27963392, count: 4096 00000080:00200000:0.0:1789760779.471316:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27963392, bytes: 4096 00000080:00200000:0.0:1789760779.471323:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.471330:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.471331:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27963392, 27967488) 0 0 00000080:00200000:0.0:1789760779.471335:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.471347:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6827, 6827] 00020000:00200000:0.0:1789760779.471353:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27963392 chunk: [27963392, 27967488) 27967488, 4096 00020000:00200000:0.0:1789760779.471357:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.471384:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6827, 6827] 00000008:00010000:0.0:1789760779.471397:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.471402:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760779.471410:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27963392, 27967488) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.471421:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27963392=0x1aab000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.471467:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.471469:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27967488 00000020:00010000:0.0:1789760779.471476:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.471479:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.471482:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.471485:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.471487:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.471494:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.471499:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.471513:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000008:00100000:0.0:1789760779.471731:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149180 - object 0x240000400:5 offset 27963392<>27967488 00000100:00100000:1.0:1789760779.471788:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512737152:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760779.522151:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974146 gen 3 00000008:00010000:1.0:1789760779.522187:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760779.522194:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.522203:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.522223:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760779.522245:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512737152:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760779.522345:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6827, 6827] 00000080:00200000:0.0:1789760779.522356:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.522371:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.522384:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27967488 i_blocks 55303 00000020:00200000:0.0:1789760779.522387:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.522396:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.522423:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27967488, count: 4096 00000080:00200000:0.0:1789760779.522429:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27967488, bytes: 4096 00000080:00200000:0.0:1789760779.522436:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.522443:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.522445:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27967488, 27971584) 0 0 00000080:00200000:0.0:1789760779.522448:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.522462:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6828, 6828] 00020000:00200000:0.0:1789760779.522468:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27967488 chunk: [27967488, 27971584) 27971584, 4096 00020000:00200000:0.0:1789760779.522472:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.522487:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6828, 6828] 00000008:00010000:0.0:1789760779.522500:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.522505:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760779.522514:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27967488, 27971584) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.522523:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27967488=0x1aac000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.522632:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.522636:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27971584 00000020:00010000:0.0:1789760779.522643:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.522646:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.522649:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.522653:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.522656:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.522662:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.522668:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.522682:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc000 00000008:00100000:0.0:1789760779.522759:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148000 - object 0x240000400:5 offset 27967488<>27971584 00000100:00100000:1.0:1789760779.522854:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512737280:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760779.576661:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974147 gen 3 00000008:00010000:1.0:1789760779.576698:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760779.576704:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.576711:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.576771:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760779.576798:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512737280:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760779.576923:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6828, 6828] 00000080:00200000:0.0:1789760779.576934:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.576939:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.576951:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27971584 i_blocks 55303 00000020:00200000:0.0:1789760779.576954:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.576962:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.576986:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27971584, count: 4096 00000080:00200000:0.0:1789760779.576992:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27971584, bytes: 4096 00000080:00200000:0.0:1789760779.576999:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.577006:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.577008:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27971584, 27975680) 0 0 00000080:00200000:0.0:1789760779.577066:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.577081:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6829, 6829] 00020000:00200000:0.0:1789760779.577088:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27971584 chunk: [27971584, 27975680) 27975680, 4096 00020000:00200000:0.0:1789760779.577091:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.577106:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6829, 6829] 00000008:00010000:0.0:1789760779.577119:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.577123:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760779.577131:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27971584, 27975680) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.577149:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27971584=0x1aad000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.577186:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.577188:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27975680 00000020:00010000:0.0:1789760779.577194:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.577197:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.577200:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.577203:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.577205:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.577212:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.577218:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.577231:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccf00 00000008:00100000:0.0:1789760779.577363:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149c00 - object 0x240000400:5 offset 27971584<>27975680 00000100:00100000:1.0:1789760779.577480:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512737408:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760779.616121:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760779.616133:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.616151:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.616163:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760779.616185:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512737408:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760779.617122:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6829, 6829] 00000080:00200000:0.0:1789760779.617136:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.617151:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.617163:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27975680 i_blocks 55303 00000020:00200000:0.0:1789760779.617166:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.617174:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.617197:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27975680, count: 4096 00000080:00200000:0.0:1789760779.617203:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27975680, bytes: 4096 00000080:00200000:0.0:1789760779.617210:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.617218:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.617220:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27975680, 27979776) 0 0 00000080:00200000:0.0:1789760779.617224:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.617234:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6830, 6830] 00020000:00200000:0.0:1789760779.617295:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27975680 chunk: [27975680, 27979776) 27979776, 4096 00020000:00200000:0.0:1789760779.617299:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.617313:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6830, 6830] 00000008:00010000:0.0:1789760779.617325:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.617329:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760779.617339:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27975680, 27979776) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.617349:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27975680=0x1aae000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.617392:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.617395:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27979776 00000020:00010000:0.0:1789760779.617401:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.617467:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.617471:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.617475:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.617478:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.617485:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.617491:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.617507:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cca80 00000008:00100000:0.0:1789760779.617707:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514bb80 - object 0x240000400:5 offset 27975680<>27979776 00000100:00100000:1.0:1789760779.617772:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512737536:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760779.650776:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974149 gen 3 00000008:00010000:1.0:1789760779.650824:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760779.650830:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.650838:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.650925:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760779.650947:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512737536:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760779.651066:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6830, 6830] 00000080:00200000:0.0:1789760779.651078:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.651084:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.651097:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27979776 i_blocks 55303 00000020:00200000:0.0:1789760779.651100:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.651109:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.651133:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27979776, count: 4096 00000080:00200000:0.0:1789760779.651148:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27979776, bytes: 4096 00000080:00200000:0.0:1789760779.651157:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.651164:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.651167:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27979776, 27983872) 0 0 00000080:00200000:0.0:1789760779.651170:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.651342:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6831, 6831] 00020000:00200000:0.0:1789760779.651350:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27979776 chunk: [27979776, 27983872) 27983872, 4096 00020000:00200000:0.0:1789760779.651353:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.651368:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6831, 6831] 00000008:00010000:0.0:1789760779.651381:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.651386:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760779.651395:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27979776, 27983872) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.651406:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27979776=0x1aaf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.651447:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.651449:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27983872 00000020:00010000:0.0:1789760779.651456:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.651459:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.651462:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.651466:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.651468:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.651475:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.651480:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.651494:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc540 00000008:00100000:0.0:1789760779.651676:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149f80 - object 0x240000400:5 offset 27979776<>27983872 00000100:00100000:1.0:1789760779.651936:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512737664:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760779.709094:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974150 gen 3 00000008:00010000:1.0:1789760779.709129:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760779.709135:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.709157:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.709184:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760779.709207:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512737664:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760779.709324:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6831, 6831] 00000080:00200000:0.0:1789760779.709336:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.709342:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.709354:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27983872 i_blocks 55303 00000020:00200000:0.0:1789760779.709368:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.709376:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.709402:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27983872, count: 4096 00000080:00200000:0.0:1789760779.709408:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27983872, bytes: 4096 00000080:00200000:0.0:1789760779.709417:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.709425:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.709427:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27983872, 27987968) 0 0 00000080:00200000:0.0:1789760779.709431:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.709447:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6832, 6832] 00020000:00200000:0.0:1789760779.709453:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27983872 chunk: [27983872, 27987968) 27987968, 4096 00020000:00200000:0.0:1789760779.709457:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.709472:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6832, 6832] 00000008:00010000:0.0:1789760779.709485:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.709490:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760779.709509:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27983872, 27987968) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.709521:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27983872=0x1ab0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.709563:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.709566:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27987968 00000020:00010000:0.0:1789760779.709572:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.709575:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.709578:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.709581:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.709584:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.709590:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.709595:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.709607:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc600 00000008:00100000:0.0:1789760779.709681:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149500 - object 0x240000400:5 offset 27983872<>27987968 00000100:00100000:1.0:1789760779.709876:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512737792:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760779.743493:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974151 gen 3 00000008:00010000:1.0:1789760779.743532:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760779.743538:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.743548:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.743559:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760779.743584:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512737792:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760779.745961:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6832, 6832] 00000080:00200000:0.0:1789760779.745980:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.745985:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.745997:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27987968 i_blocks 55303 00000020:00200000:0.0:1789760779.746001:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.746014:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.746041:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27987968, count: 4096 00000080:00200000:0.0:1789760779.746049:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27987968, bytes: 4096 00000080:00200000:0.0:1789760779.746060:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.746067:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.746069:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27987968, 27992064) 0 0 00000080:00200000:0.0:1789760779.746073:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.746087:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6833, 6833] 00020000:00200000:0.0:1789760779.746094:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27987968 chunk: [27987968, 27992064) 27992064, 4096 00020000:00200000:0.0:1789760779.746098:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.746114:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6833, 6833] 00000008:00010000:0.0:1789760779.746128:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.746133:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760779.746152:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27987968, 27992064) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.746164:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27987968=0x1ab1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.746214:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.746217:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27992064 00000020:00010000:0.0:1789760779.746225:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.746229:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.746231:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.746235:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.746237:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.746245:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.746250:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.746267:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000008:00100000:0.0:1789760779.746350:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b100 - object 0x240000400:5 offset 27987968<>27992064 00000100:00100000:1.0:1789760779.748288:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760779.748324:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512737920:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760779.783398:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974152 gen 3 00000008:00010000:1.0:1789760779.783439:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760779.783446:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.783456:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.783479:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760779.783515:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512737920:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760779.783707:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6833, 6833] 00000080:00200000:0.0:1789760779.783719:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.783725:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.783736:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27992064 i_blocks 55303 00000020:00200000:0.0:1789760779.783740:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.783747:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.783773:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27992064, count: 4096 00000080:00200000:0.0:1789760779.783779:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27992064, bytes: 4096 00000080:00200000:0.0:1789760779.783787:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.783794:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.783796:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27992064, 27996160) 0 0 00000080:00200000:0.0:1789760779.783799:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.783812:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6834, 6834] 00020000:00200000:0.0:1789760779.783819:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27992064 chunk: [27992064, 27996160) 27996160, 4096 00020000:00200000:0.0:1789760779.783823:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.783838:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6834, 6834] 00000008:00010000:0.0:1789760779.783852:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.783857:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760779.783866:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27992064, 27996160) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.783877:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27992064=0x1ab2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.783950:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.783953:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 27996160 00000020:00010000:0.0:1789760779.783960:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.783964:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.783966:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.783970:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.783972:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.783979:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.783985:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.784000:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cccc0 00000008:00100000:0.0:1789760779.784081:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b100 - object 0x240000400:5 offset 27992064<>27996160 00000100:00100000:1.0:1789760779.784452:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512738048:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760779.816942:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974153 gen 3 00000008:00010000:1.0:1789760779.816978:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760779.816984:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.816993:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.817016:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760779.817038:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512738048:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760779.817370:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6834, 6834] 00000080:00200000:0.0:1789760779.817383:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.817390:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.817402:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 27996160 i_blocks 55303 00000020:00200000:0.0:1789760779.817406:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.817414:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.817442:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 27996160, count: 4096 00000080:00200000:0.0:1789760779.817448:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 27996160, bytes: 4096 00000080:00200000:0.0:1789760779.817456:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.817464:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.817466:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [27996160, 28000256) 0 0 00000080:00200000:0.0:1789760779.817470:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.817484:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6835, 6835] 00020000:00200000:0.0:1789760779.817491:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 27996160 chunk: [27996160, 28000256) 28000256, 4096 00020000:00200000:0.0:1789760779.817495:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.817511:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6835, 6835] 00000008:00010000:0.0:1789760779.817524:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.817529:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760779.817538:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [27996160, 28000256) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.817547:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=27996160=0x1ab3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.817591:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.817594:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28000256 00000020:00010000:0.0:1789760779.817601:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.817604:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.817607:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.817610:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.817612:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.817619:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.817625:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.817639:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc900 00000008:00100000:0.0:1789760779.817711:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149500 - object 0x240000400:5 offset 27996160<>28000256 00000100:00100000:1.0:1789760779.817794:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760779.817807:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512738176:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760779.847674:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974154 gen 3 00000008:00010000:1.0:1789760779.847710:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760779.847716:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.847726:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.847795:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760779.847814:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512738176:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760779.850288:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6835, 6835] 00000080:00200000:0.0:1789760779.850305:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.850312:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.850324:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28000256 i_blocks 55303 00000020:00200000:0.0:1789760779.850329:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.850340:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.850365:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28000256, count: 4096 00000080:00200000:0.0:1789760779.850372:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28000256, bytes: 4096 00000080:00200000:0.0:1789760779.850381:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.850388:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.850390:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28000256, 28004352) 0 0 00000080:00200000:0.0:1789760779.850394:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.850406:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6836, 6836] 00020000:00200000:0.0:1789760779.850412:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28000256 chunk: [28000256, 28004352) 28004352, 4096 00020000:00200000:0.0:1789760779.850416:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.850431:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6836, 6836] 00000008:00010000:0.0:1789760779.850445:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.850450:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760779.850459:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28000256, 28004352) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.850469:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28000256=0x1ab4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.850519:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.850522:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28004352 00000020:00010000:0.0:1789760779.850598:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.850602:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.850605:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.850608:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.850610:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.850617:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.850623:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.850646:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000008:00100000:0.0:1789760779.850725:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149f80 - object 0x240000400:5 offset 28000256<>28004352 00000100:00100000:1.0:1789760779.850810:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512738304:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760779.881189:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974155 gen 3 00000008:00010000:1.0:1789760779.881225:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760779.881231:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.881240:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.881264:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760779.881286:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512738304:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760779.881386:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6836, 6836] 00000080:00200000:2.0:1789760779.881399:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.881404:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.881416:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28004352 i_blocks 55303 00000020:00200000:2.0:1789760779.881419:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.881427:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.881452:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28004352, count: 4096 00000080:00200000:2.0:1789760779.881458:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28004352, bytes: 4096 00000080:00200000:2.0:1789760779.881465:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.881472:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.881474:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28004352, 28008448) 0 0 00000080:00200000:2.0:1789760779.881478:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.881490:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6837, 6837] 00020000:00200000:2.0:1789760779.881497:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28004352 chunk: [28004352, 28008448) 28008448, 4096 00020000:00200000:2.0:1789760779.881501:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.881514:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6837, 6837] 00000008:00010000:2.0:1789760779.881526:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.881531:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760779.881540:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28004352, 28008448) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.881550:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28004352=0x1ab5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.881602:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.881605:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28008448 00000020:00010000:2.0:1789760779.881612:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.881615:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.881618:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.881621:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.881624:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.881630:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.881645:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760779.881658:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760779.881739:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 28004352<>28008448 00000100:00100000:2.0:1789760779.881778:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760779.881792:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512738432:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760779.914958:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974156 gen 3 00000008:00010000:2.0:1789760779.914992:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:2.0:1789760779.914999:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760779.915006:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760779.915080:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:2.0:1789760779.915103:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512738432:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760779.915298:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6837, 6837] 00000080:00200000:0.0:1789760779.915311:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.915316:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.915327:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28008448 i_blocks 55303 00000020:00200000:0.0:1789760779.915330:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.915338:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.915365:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28008448, count: 4096 00000080:00200000:0.0:1789760779.915371:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28008448, bytes: 4096 00000080:00200000:0.0:1789760779.915379:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.915386:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.915388:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28008448, 28012544) 0 0 00000080:00200000:0.0:1789760779.915392:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.915404:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6838, 6838] 00020000:00200000:0.0:1789760779.915412:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28008448 chunk: [28008448, 28012544) 28012544, 4096 00020000:00200000:0.0:1789760779.915415:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.915431:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6838, 6838] 00000008:00010000:0.0:1789760779.915444:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.915449:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760779.915458:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28008448, 28012544) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.915468:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28008448=0x1ab6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.915511:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.915525:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28012544 00000020:00010000:0.0:1789760779.915532:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.915536:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.915539:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.915543:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.915545:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.915552:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.915558:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.915673:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccb40 00000008:00100000:0.0:1789760779.915811:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514bb80 - object 0x240000400:5 offset 28008448<>28012544 00000100:00100000:1.0:1789760779.916544:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512738560:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760779.954659:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760779.954669:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760779.954677:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760779.954698:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760779.954721:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512738560:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760779.955212:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6838, 6838] 00000080:00200000:0.0:1789760779.955225:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760779.955231:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760779.955242:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28012544 i_blocks 55303 00000020:00200000:0.0:1789760779.955246:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760779.955254:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760779.955290:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28012544, count: 4096 00000080:00200000:0.0:1789760779.955297:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28012544, bytes: 4096 00000080:00200000:0.0:1789760779.955306:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760779.955314:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760779.955316:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28012544, 28016640) 0 0 00000080:00200000:0.0:1789760779.955319:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760779.955332:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6839, 6839] 00020000:00200000:0.0:1789760779.955338:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28012544 chunk: [28012544, 28016640) 28016640, 4096 00020000:00200000:0.0:1789760779.955341:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760779.955356:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6839, 6839] 00000008:00010000:0.0:1789760779.955369:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760779.955373:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760779.955384:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28012544, 28016640) append 0 append lockpos 0 00000080:00200000:0.0:1789760779.955394:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28012544=0x1ab7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760779.955453:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760779.955457:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28016640 00000020:00010000:0.0:1789760779.955463:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760779.955466:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.955468:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760779.955471:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760779.955474:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760779.955480:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760779.955485:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760779.955500:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc300 00000008:00100000:0.0:1789760779.955582:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149c00 - object 0x240000400:5 offset 28012544<>28016640 00000100:00100000:1.0:1789760779.955865:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760779.955882:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512738688:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760779.990151:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974158 gen 3 00000008:00010000:0.0:1789760779.990201:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:0.0:1789760779.990208:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760779.990216:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760779.990238:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:0.0:1789760779.990261:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512738688:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760779.990352:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6839, 6839] 00000080:00200000:2.0:1789760779.990365:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760779.990370:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760779.990385:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28016640 i_blocks 55303 00000020:00200000:2.0:1789760779.990389:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760779.990398:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760779.990424:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28016640, count: 4096 00000080:00200000:2.0:1789760779.990431:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28016640, bytes: 4096 00000080:00200000:2.0:1789760779.990439:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760779.990447:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760779.990448:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28016640, 28020736) 0 0 00000080:00200000:2.0:1789760779.990452:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760779.990466:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6840, 6840] 00020000:00200000:2.0:1789760779.990473:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28016640 chunk: [28016640, 28020736) 28020736, 4096 00020000:00200000:2.0:1789760779.990477:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760779.990491:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6840, 6840] 00000008:00010000:2.0:1789760779.990505:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760779.990509:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760779.990519:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28016640, 28020736) append 0 append lockpos 0 00000080:00200000:2.0:1789760779.990541:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28016640=0x1ab8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760779.990588:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760779.990590:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28020736 00000020:00010000:2.0:1789760779.990598:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760779.990601:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.990604:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760779.990607:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760779.990609:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760779.990615:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760779.990620:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760779.990634:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760779.990710:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 28016640<>28020736 00000100:00100000:3.0:1789760779.990940:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512738816:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760780.025736:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:2.0:1789760780.025748:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760780.025757:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760780.025778:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:2.0:1789760780.025803:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512738816:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760780.025931:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6840, 6840] 00000080:00200000:0.0:1789760780.025945:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760780.025951:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760780.026000:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28020736 i_blocks 55303 00000020:00200000:0.0:1789760780.026004:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760780.026013:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760780.026043:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28020736, count: 4096 00000080:00200000:0.0:1789760780.026050:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28020736, bytes: 4096 00000080:00200000:0.0:1789760780.026058:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760780.026065:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760780.026067:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28020736, 28024832) 0 0 00000080:00200000:0.0:1789760780.026071:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760780.026084:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6841, 6841] 00020000:00200000:0.0:1789760780.026091:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28020736 chunk: [28020736, 28024832) 28024832, 4096 00020000:00200000:0.0:1789760780.026095:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760780.026109:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6841, 6841] 00000008:00010000:0.0:1789760780.026122:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760780.026127:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760780.026147:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28020736, 28024832) append 0 append lockpos 0 00000080:00200000:0.0:1789760780.026159:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28020736=0x1ab9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760780.026204:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760780.026207:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28024832 00000020:00010000:0.0:1789760780.026214:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760780.026218:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760780.026220:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760780.026223:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760780.026226:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760780.026233:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760780.026238:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760780.026251:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc300 00000008:00100000:0.0:1789760780.026328:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149c00 - object 0x240000400:5 offset 28020736<>28024832 00000100:00100000:0.0:1789760780.026371:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760780.026386:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512738944:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760780.058430:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974160 gen 3 00000008:00010000:0.0:1789760780.058481:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:0.0:1789760780.058489:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760780.058499:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760780.058524:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:0.0:1789760780.058549:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512738944:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760780.058644:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6841, 6841] 00000080:00200000:2.0:1789760780.058656:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.058661:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.058674:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28024832 i_blocks 55303 00000020:00200000:2.0:1789760780.058677:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.058684:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.058710:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28024832, count: 4096 00000080:00200000:2.0:1789760780.058715:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28024832, bytes: 4096 00000080:00200000:2.0:1789760780.058723:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.058730:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.058732:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28024832, 28028928) 0 0 00000080:00200000:2.0:1789760780.058735:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.058748:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6842, 6842] 00020000:00200000:2.0:1789760780.058754:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28024832 chunk: [28024832, 28028928) 28028928, 4096 00020000:00200000:2.0:1789760780.058758:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.058772:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6842, 6842] 00000008:00010000:2.0:1789760780.058784:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.058799:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760780.058816:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28024832, 28028928) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.058827:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28024832=0x1aba000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.058868:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.058870:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28028928 00000020:00010000:2.0:1789760780.058876:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.058879:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.058882:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.058885:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.058887:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.058894:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.059520:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.059595:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760780.059674:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 28024832<>28028928 00000100:00100000:3.0:1789760780.059748:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512739072:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760780.061202:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512739328:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1789760780.061341:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e6cf7267-9b19-462a-a263-14115df15dab:2358:1876699512739200:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1789760780.061385:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2358:1876699512739456:192.168.203.156@tcp:400:kworker.0 00010000:00010000:2.0:1789760780.061477:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760780.065116:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2358:1876699512739456:192.168.203.156@tcp:400:kworker.0 00000100:00100000:1.0:1789760780.066103:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512739328:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1789760780.068998:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e6cf7267-9b19-462a-a263-14115df15dab:2358:1876699512739200:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760780.109644:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974161 gen 3 00000008:00010000:3.0:1789760780.109679:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760780.109685:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.109694:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.109710:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760780.109736:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512739072:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760780.109829:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6842, 6842] 00000080:00200000:2.0:1789760780.109842:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.109848:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.109858:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28028928 i_blocks 55303 00000020:00200000:2.0:1789760780.109862:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.109870:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.109895:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28028928, count: 4096 00000080:00200000:2.0:1789760780.109923:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28028928, bytes: 4096 00000080:00200000:2.0:1789760780.109931:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.109938:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.109940:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28028928, 28033024) 0 0 00000080:00200000:2.0:1789760780.109943:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.109956:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6843, 6843] 00020000:00200000:2.0:1789760780.109964:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28028928 chunk: [28028928, 28033024) 28033024, 4096 00020000:00200000:2.0:1789760780.109967:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.109983:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6843, 6843] 00000008:00010000:2.0:1789760780.109997:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.110001:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760780.110010:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28028928, 28033024) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.110021:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28028928=0x1abb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.110062:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.110064:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28033024 00000020:00010000:2.0:1789760780.110070:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.110074:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.110076:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.110079:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.110082:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.110088:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.110094:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.110107:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760780.110189:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 28028928<>28033024 00000100:00100000:3.0:1789760780.110280:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512739584:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760780.146392:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974162 gen 3 00000008:00010000:3.0:1789760780.146437:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760780.146444:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.146453:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.146508:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760780.146533:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512739584:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760780.146607:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6843, 6843] 00000080:00200000:2.0:1789760780.146620:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.146625:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.146635:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28033024 i_blocks 55303 00000020:00200000:2.0:1789760780.146638:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.146645:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.146825:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28033024, count: 4096 00000080:00200000:2.0:1789760780.146834:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28033024, bytes: 4096 00000080:00200000:2.0:1789760780.146843:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.146850:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.146853:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28033024, 28037120) 0 0 00000080:00200000:2.0:1789760780.146857:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.146870:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6844, 6844] 00020000:00200000:2.0:1789760780.146878:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28033024 chunk: [28033024, 28037120) 28037120, 4096 00020000:00200000:2.0:1789760780.146882:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.146898:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6844, 6844] 00000008:00010000:2.0:1789760780.146934:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.146938:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760780.146946:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28033024, 28037120) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.146957:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28033024=0x1abc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.146998:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.147001:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28037120 00000020:00010000:2.0:1789760780.147008:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.147011:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.147013:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.147017:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.147019:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.147025:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.147033:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.147048:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760780.147205:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 28033024<>28037120 00000100:00100000:3.0:1789760780.147264:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512739712:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760780.185609:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760780.185621:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.185629:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.185646:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760780.185670:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512739712:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760780.185745:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6844, 6844] 00000080:00200000:2.0:1789760780.185759:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.185764:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.185776:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28037120 i_blocks 55303 00000020:00200000:2.0:1789760780.185779:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.185787:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.185827:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28037120, count: 4096 00000080:00200000:2.0:1789760780.185834:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28037120, bytes: 4096 00000080:00200000:2.0:1789760780.185840:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.185847:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.185849:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28037120, 28041216) 0 0 00000080:00200000:2.0:1789760780.185853:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.185866:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6845, 6845] 00020000:00200000:2.0:1789760780.185872:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28037120 chunk: [28037120, 28041216) 28041216, 4096 00020000:00200000:2.0:1789760780.185876:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.185890:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6845, 6845] 00000008:00010000:2.0:1789760780.185929:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.185934:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760780.185943:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28037120, 28041216) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.185954:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28037120=0x1abd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.185993:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.185995:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28041216 00000020:00010000:2.0:1789760780.186001:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.186004:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.186007:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.186010:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.186013:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.186019:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.186025:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.186037:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760780.186114:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 28037120<>28041216 00000100:00100000:3.0:1789760780.186269:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512739840:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760780.217223:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974164 gen 3 00000008:00010000:3.0:1789760780.217271:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760780.217277:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.217285:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.217401:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760780.217427:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512739840:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760780.217439:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6845, 6845] 00000080:00200000:2.0:1789760780.217455:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.217461:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.217474:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28041216 i_blocks 55303 00000020:00200000:2.0:1789760780.217478:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.217488:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.217510:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28041216, count: 4096 00000080:00200000:2.0:1789760780.217516:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28041216, bytes: 4096 00000080:00200000:2.0:1789760780.217523:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.217531:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.217533:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28041216, 28045312) 0 0 00000080:00200000:2.0:1789760780.217537:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.217550:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6846, 6846] 00020000:00200000:2.0:1789760780.217557:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28041216 chunk: [28041216, 28045312) 28045312, 4096 00020000:00200000:2.0:1789760780.217561:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.217576:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6846, 6846] 00000008:00010000:2.0:1789760780.217590:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.217595:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760780.217604:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28041216, 28045312) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.217613:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28041216=0x1abe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.217655:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.217657:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28045312 00000020:00010000:2.0:1789760780.217664:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.217780:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.217785:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.217788:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.217791:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.217799:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.217806:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.217819:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760780.217891:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 28041216<>28045312 00000100:00100000:3.0:1789760780.218091:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512739968:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760780.276533:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760780.276544:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.276553:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.276576:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760780.276602:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512739968:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760780.276798:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6846, 6846] 00000080:00200000:0.0:1789760780.276814:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760780.276819:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760780.276833:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28045312 i_blocks 55303 00000020:00200000:0.0:1789760780.276839:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760780.276847:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760780.276880:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28045312, count: 4096 00000080:00200000:0.0:1789760780.276886:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28045312, bytes: 4096 00000080:00200000:0.0:1789760780.276897:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760780.276935:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760780.276937:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28045312, 28049408) 0 0 00000080:00200000:0.0:1789760780.276941:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760780.276956:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6847, 6847] 00020000:00200000:0.0:1789760780.276965:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28045312 chunk: [28045312, 28049408) 28049408, 4096 00020000:00200000:0.0:1789760780.276969:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760780.276988:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6847, 6847] 00000008:00010000:0.0:1789760780.277015:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760780.277020:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760780.277031:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28045312, 28049408) append 0 append lockpos 0 00000080:00200000:0.0:1789760780.277044:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28045312=0x1abf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760780.277092:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760780.277094:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28049408 00000020:00010000:0.0:1789760780.277101:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760780.277104:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760780.277107:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760780.277110:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760780.277112:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760780.277118:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760780.277124:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760780.277145:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc300 00000008:00100000:0.0:1789760780.277235:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149c00 - object 0x240000400:5 offset 28045312<>28049408 00000100:00100000:0.0:1789760780.277274:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760780.277287:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512740096:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760780.340298:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974166 gen 3 00000008:00010000:0.0:1789760780.340345:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:0.0:1789760780.340352:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760780.340360:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760780.340383:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:0.0:1789760780.340417:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512740096:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760780.340427:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6847, 6847] 00000080:00200000:3.0:1789760780.340443:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760780.340449:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760780.340461:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28049408 i_blocks 55303 00000020:00200000:3.0:1789760780.340465:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760780.340473:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760780.340499:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28049408, count: 4096 00000080:00200000:3.0:1789760780.340504:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28049408, bytes: 4096 00000080:00200000:3.0:1789760780.340512:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760780.340520:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760780.340522:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28049408, 28053504) 0 0 00000080:00200000:3.0:1789760780.340622:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760780.340637:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6848, 6848] 00020000:00200000:3.0:1789760780.340644:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28049408 chunk: [28049408, 28053504) 28053504, 4096 00020000:00200000:3.0:1789760780.340649:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760780.340665:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6848, 6848] 00000008:00010000:3.0:1789760780.340681:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760780.340686:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760780.340695:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28049408, 28053504) append 0 append lockpos 0 00000080:00200000:3.0:1789760780.340706:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28049408=0x1ac0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760780.340749:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760780.340752:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28053504 00000020:00010000:3.0:1789760780.340758:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760780.340760:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760780.340763:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760780.340766:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760780.340768:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760780.340774:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760780.340779:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760780.340883:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760780.341082:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac380 - object 0x240000400:5 offset 28049408<>28053504 00000100:00100000:2.0:1789760780.341266:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512740224:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760780.381072:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974167 gen 3 00000008:00010000:3.0:1789760780.381175:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:3.0:1789760780.381182:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.381191:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.381243:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:3.0:1789760780.381265:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512740224:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760780.381335:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6848, 6848] 00000080:00200000:2.0:1789760780.381348:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.381353:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.381364:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28053504 i_blocks 55303 00000020:00200000:2.0:1789760780.381367:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.381374:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.381402:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28053504, count: 4096 00000080:00200000:2.0:1789760780.381407:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28053504, bytes: 4096 00000080:00200000:2.0:1789760780.381415:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.381422:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.381424:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28053504, 28057600) 0 0 00000080:00200000:2.0:1789760780.381427:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.381440:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6849, 6849] 00020000:00200000:2.0:1789760780.381448:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28053504 chunk: [28053504, 28057600) 28057600, 4096 00020000:00200000:2.0:1789760780.381451:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.381466:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6849, 6849] 00000008:00010000:2.0:1789760780.381479:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.381483:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760780.381492:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28053504, 28057600) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.381502:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28053504=0x1ac1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.381616:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.381619:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28057600 00000020:00010000:2.0:1789760780.381626:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.381628:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.381631:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.381633:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.381635:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.381640:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.381645:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.381701:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760780.381765:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 28053504<>28057600 00000100:00100000:3.0:1789760780.381820:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512740352:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760780.483526:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974168 gen 3 00000008:00010000:3.0:1789760780.483560:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760780.483566:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.483574:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.483592:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760780.483629:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512740352:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760780.484141:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6849, 6849] 00000080:00200000:2.0:1789760780.484155:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.484160:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.484171:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28057600 i_blocks 55303 00000020:00200000:2.0:1789760780.484175:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.484182:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.484207:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28057600, count: 4096 00000080:00200000:2.0:1789760780.484212:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28057600, bytes: 4096 00000080:00200000:2.0:1789760780.484220:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.484227:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.484228:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28057600, 28061696) 0 0 00000080:00200000:2.0:1789760780.484232:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.484245:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6850, 6850] 00020000:00200000:2.0:1789760780.484251:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28057600 chunk: [28057600, 28061696) 28061696, 4096 00020000:00200000:2.0:1789760780.484255:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.484269:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6850, 6850] 00000008:00010000:2.0:1789760780.484282:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.484287:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760780.484295:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28057600, 28061696) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.484304:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28057600=0x1ac2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.484344:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.484347:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28061696 00000020:00010000:2.0:1789760780.484353:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.484356:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.484359:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.484362:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.484365:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.484371:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.484377:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.484391:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760780.484465:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 28057600<>28061696 00000100:00100000:3.0:1789760780.484625:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512740480:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760780.541999:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760780.542024:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.542035:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.542050:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760780.542074:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512740480:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760780.542228:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6850, 6850] 00000080:00200000:2.0:1789760780.542241:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.542246:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.542259:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28061696 i_blocks 55303 00000020:00200000:2.0:1789760780.542262:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.542271:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.542300:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28061696, count: 4096 00000080:00200000:2.0:1789760780.542306:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28061696, bytes: 4096 00000080:00200000:2.0:1789760780.542313:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.542321:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.542323:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28061696, 28065792) 0 0 00000080:00200000:2.0:1789760780.542327:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.542343:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6851, 6851] 00020000:00200000:2.0:1789760780.542350:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28061696 chunk: [28061696, 28065792) 28065792, 4096 00020000:00200000:2.0:1789760780.542353:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.542369:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6851, 6851] 00000008:00010000:2.0:1789760780.542385:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.542391:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760780.542401:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28061696, 28065792) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.542413:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28061696=0x1ac3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.542464:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.542467:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28065792 00000020:00010000:2.0:1789760780.542474:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.542477:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.542479:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.542482:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.542485:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.542492:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.542497:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.542511:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760780.542586:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 28061696<>28065792 00000100:00100000:3.0:1789760780.543161:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760780.543178:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512740608:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760780.578196:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974170 gen 3 00000008:00010000:3.0:1789760780.578239:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760780.578247:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.578256:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.578275:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760780.578297:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512740608:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760780.578381:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6851, 6851] 00000080:00200000:2.0:1789760780.578393:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.578398:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.578410:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28065792 i_blocks 55303 00000020:00200000:2.0:1789760780.578413:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.578421:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.578446:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28065792, count: 4096 00000080:00200000:2.0:1789760780.578452:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28065792, bytes: 4096 00000080:00200000:2.0:1789760780.578460:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.578466:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.578468:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28065792, 28069888) 0 0 00000080:00200000:2.0:1789760780.578471:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.578483:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6852, 6852] 00020000:00200000:2.0:1789760780.578490:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28065792 chunk: [28065792, 28069888) 28069888, 4096 00020000:00200000:2.0:1789760780.578493:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.578507:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6852, 6852] 00000008:00010000:2.0:1789760780.578520:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.578590:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760780.578601:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28065792, 28069888) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.578612:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28065792=0x1ac4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.578654:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.578657:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28069888 00000020:00010000:2.0:1789760780.578663:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.578666:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.578669:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.578672:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.578675:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.578681:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.578687:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.578700:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760780.578769:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 28065792<>28069888 00000100:00100000:3.0:1789760780.578859:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512740736:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760780.613968:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974171 gen 3 00000008:00010000:3.0:1789760780.614015:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760780.614022:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.614031:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.614050:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760780.614071:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512740736:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760780.614207:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6852, 6852] 00000080:00200000:2.0:1789760780.614219:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.614224:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.614235:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28069888 i_blocks 55303 00000020:00200000:2.0:1789760780.614239:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.614246:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.614271:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28069888, count: 4096 00000080:00200000:2.0:1789760780.614277:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28069888, bytes: 4096 00000080:00200000:2.0:1789760780.614284:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.614292:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.614294:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28069888, 28073984) 0 0 00000080:00200000:2.0:1789760780.614298:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.614310:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6853, 6853] 00020000:00200000:2.0:1789760780.614317:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28069888 chunk: [28069888, 28073984) 28073984, 4096 00020000:00200000:2.0:1789760780.614321:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.614336:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6853, 6853] 00000008:00010000:2.0:1789760780.614351:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.614356:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760780.614365:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28069888, 28073984) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.614376:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28069888=0x1ac5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.614424:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.614427:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28073984 00000020:00010000:2.0:1789760780.614433:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.614436:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.614439:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.614442:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.614446:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.614452:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.614457:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.614470:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760780.614533:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 28069888<>28073984 00000100:00100000:3.0:1789760780.614588:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512740864:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760780.654772:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974172 gen 3 00000008:00010000:3.0:1789760780.654925:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760780.654931:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.654939:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.654956:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760780.654979:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512740864:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760780.655063:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6853, 6853] 00000080:00200000:2.0:1789760780.655075:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.655081:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.655092:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28073984 i_blocks 55303 00000020:00200000:2.0:1789760780.655095:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.655103:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.655126:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28073984, count: 4096 00000080:00200000:2.0:1789760780.655142:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28073984, bytes: 4096 00000080:00200000:2.0:1789760780.655150:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.655157:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.655159:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28073984, 28078080) 0 0 00000080:00200000:2.0:1789760780.655163:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.655177:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6854, 6854] 00020000:00200000:2.0:1789760780.655184:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28073984 chunk: [28073984, 28078080) 28078080, 4096 00020000:00200000:2.0:1789760780.655188:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.655202:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6854, 6854] 00000008:00010000:2.0:1789760780.655215:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.655220:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760780.655228:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28073984, 28078080) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.655237:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28073984=0x1ac6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.655279:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.655282:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28078080 00000020:00010000:2.0:1789760780.655288:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.655291:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.655294:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.655297:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.655301:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.655307:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.655313:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.655326:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760780.655402:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 28073984<>28078080 00000100:00100000:3.0:1789760780.655457:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512740992:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760780.690420:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974173 gen 3 00000008:00010000:3.0:1789760780.690460:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760780.690466:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.690476:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.690649:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760780.690673:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512740992:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760780.690747:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6854, 6854] 00000080:00200000:2.0:1789760780.690760:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.690765:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.690777:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28078080 i_blocks 55303 00000020:00200000:2.0:1789760780.690781:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.690788:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.690878:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28078080, count: 4096 00000080:00200000:2.0:1789760780.690885:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28078080, bytes: 4096 00000080:00200000:2.0:1789760780.690893:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.690917:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.690919:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28078080, 28082176) 0 0 00000080:00200000:2.0:1789760780.690923:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.690935:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6855, 6855] 00020000:00200000:2.0:1789760780.690941:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28078080 chunk: [28078080, 28082176) 28082176, 4096 00020000:00200000:2.0:1789760780.690945:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.690958:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6855, 6855] 00000008:00010000:2.0:1789760780.690969:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.690973:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760780.690981:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28078080, 28082176) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.690991:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28078080=0x1ac7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.691029:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.691032:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28082176 00000020:00010000:2.0:1789760780.691037:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.691040:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.691043:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.691046:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.691049:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.691054:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.691059:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.691072:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760780.691151:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 28078080<>28082176 00000100:00100000:3.0:1789760780.691208:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512741120:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760780.739395:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760780.739404:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.739413:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.739433:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760780.739459:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512741120:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760780.739542:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6855, 6855] 00000080:00200000:2.0:1789760780.739555:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.739560:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.739571:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28082176 i_blocks 55303 00000020:00200000:2.0:1789760780.739575:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.739584:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.739609:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28082176, count: 4096 00000080:00200000:2.0:1789760780.739615:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28082176, bytes: 4096 00000080:00200000:2.0:1789760780.739623:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.739630:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.739632:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28082176, 28086272) 0 0 00000080:00200000:2.0:1789760780.739635:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.739648:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6856, 6856] 00020000:00200000:2.0:1789760780.739655:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28082176 chunk: [28082176, 28086272) 28086272, 4096 00020000:00200000:2.0:1789760780.739659:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.739674:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6856, 6856] 00000008:00010000:2.0:1789760780.739688:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.739693:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760780.739701:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28082176, 28086272) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.739711:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28082176=0x1ac8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.739766:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.739769:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28086272 00000020:00010000:2.0:1789760780.739776:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.739779:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.739782:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.739786:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.739789:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.739795:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.739801:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.739815:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760780.740068:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 28082176<>28086272 00000100:00100000:3.0:1789760780.740274:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512741248:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760780.784316:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974175 gen 3 00000008:00010000:3.0:1789760780.784363:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760780.784370:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.784377:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.784397:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760780.784421:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512741248:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760780.784559:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6856, 6856] 00000080:00200000:2.0:1789760780.784571:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.784576:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.784586:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28086272 i_blocks 55303 00000020:00200000:2.0:1789760780.784589:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.784597:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.784622:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28086272, count: 4096 00000080:00200000:2.0:1789760780.784629:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28086272, bytes: 4096 00000080:00200000:2.0:1789760780.784637:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.784644:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.784646:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28086272, 28090368) 0 0 00000080:00200000:2.0:1789760780.784650:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.784662:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6857, 6857] 00020000:00200000:2.0:1789760780.784668:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28086272 chunk: [28086272, 28090368) 28090368, 4096 00020000:00200000:2.0:1789760780.784672:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.784687:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6857, 6857] 00000008:00010000:2.0:1789760780.784700:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.784705:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760780.784726:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28086272, 28090368) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.784738:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28086272=0x1ac9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.784778:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.784781:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28090368 00000020:00010000:2.0:1789760780.784787:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.784791:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.784793:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.784796:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.784800:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.784806:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.784812:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.784824:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760780.785186:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 28086272<>28090368 00000100:00100000:3.0:1789760780.785246:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512741376:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760780.830353:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760780.830366:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.830376:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.830397:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760780.830423:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512741376:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760780.830503:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6857, 6857] 00000080:00200000:2.0:1789760780.830517:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.830522:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.830534:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28090368 i_blocks 55303 00000020:00200000:2.0:1789760780.830538:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.830545:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.830574:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28090368, count: 4096 00000080:00200000:2.0:1789760780.830580:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28090368, bytes: 4096 00000080:00200000:2.0:1789760780.830588:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.830595:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.830597:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28090368, 28094464) 0 0 00000080:00200000:2.0:1789760780.830601:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.830614:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6858, 6858] 00020000:00200000:2.0:1789760780.830621:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28090368 chunk: [28090368, 28094464) 28094464, 4096 00020000:00200000:2.0:1789760780.830625:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.830640:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6858, 6858] 00000008:00010000:2.0:1789760780.830664:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.830669:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760780.830678:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28090368, 28094464) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.830689:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28090368=0x1aca000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.830731:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.830733:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28094464 00000020:00010000:2.0:1789760780.830740:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.830743:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.830747:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.830750:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.830753:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.830760:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.830765:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.830778:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760780.830853:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 28090368<>28094464 00000100:00100000:3.0:1789760780.831051:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512741504:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760780.885757:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974177 gen 3 00000008:00010000:3.0:1789760780.885808:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760780.885815:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.885823:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.885892:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760780.885927:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512741504:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760780.886004:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6858, 6858] 00000080:00200000:2.0:1789760780.886017:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.886023:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.886034:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28094464 i_blocks 55303 00000020:00200000:2.0:1789760780.886037:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.886045:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.886071:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28094464, count: 4096 00000080:00200000:2.0:1789760780.886078:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28094464, bytes: 4096 00000080:00200000:2.0:1789760780.886088:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.886095:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.886097:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28094464, 28098560) 0 0 00000080:00200000:2.0:1789760780.886101:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.886189:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6859, 6859] 00020000:00200000:2.0:1789760780.886253:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28094464 chunk: [28094464, 28098560) 28098560, 4096 00020000:00200000:2.0:1789760780.886258:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.886275:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6859, 6859] 00000008:00010000:2.0:1789760780.886288:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.886293:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760780.886302:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28094464, 28098560) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.886312:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28094464=0x1acb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.886359:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.886361:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28098560 00000020:00010000:2.0:1789760780.886368:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.886371:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.886373:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.886376:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.886379:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.886385:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.886391:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.886407:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760780.886487:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 28094464<>28098560 00000100:00100000:3.0:1789760780.886544:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512741632:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760780.944504:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974178 gen 3 00000008:00010000:3.0:1789760780.944700:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760780.944707:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760780.944717:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760780.944746:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760780.944771:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512741632:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760780.944967:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6859, 6859] 00000080:00200000:2.0:1789760780.944982:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760780.944987:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760780.944998:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28098560 i_blocks 55303 00000020:00200000:2.0:1789760780.945002:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760780.945010:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760780.945035:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28098560, count: 4096 00000080:00200000:2.0:1789760780.945042:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28098560, bytes: 4096 00000080:00200000:2.0:1789760780.945053:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760780.945061:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760780.945063:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28098560, 28102656) 0 0 00000080:00200000:2.0:1789760780.945066:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760780.945079:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6860, 6860] 00020000:00200000:2.0:1789760780.945085:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28098560 chunk: [28098560, 28102656) 28102656, 4096 00020000:00200000:2.0:1789760780.945089:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760780.945104:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6860, 6860] 00000008:00010000:2.0:1789760780.945118:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760780.945122:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760780.945168:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28098560, 28102656) append 0 append lockpos 0 00000080:00200000:2.0:1789760780.945180:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28098560=0x1acc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760780.945228:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760780.945230:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28102656 00000020:00010000:2.0:1789760780.945237:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760780.945240:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.945244:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760780.945247:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760780.945251:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760780.945258:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760780.945263:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760780.945276:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760780.945360:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 28098560<>28102656 00000100:00100000:3.0:1789760780.948274:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512741760:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760781.002527:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974179 gen 3 00000008:00010000:3.0:1789760781.002568:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760781.002575:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.002585:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.002605:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760781.002629:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512741760:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760781.002725:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6860, 6860] 00000080:00200000:2.0:1789760781.002740:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.002745:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.002758:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28102656 i_blocks 55303 00000020:00200000:2.0:1789760781.002762:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.002771:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.002796:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28102656, count: 4096 00000080:00200000:2.0:1789760781.002803:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28102656, bytes: 4096 00000080:00200000:2.0:1789760781.002811:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.002819:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.002821:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28102656, 28106752) 0 0 00000080:00200000:2.0:1789760781.002825:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.002837:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6861, 6861] 00020000:00200000:2.0:1789760781.002844:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28102656 chunk: [28102656, 28106752) 28106752, 4096 00020000:00200000:2.0:1789760781.002848:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.002864:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6861, 6861] 00000008:00010000:2.0:1789760781.002878:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.002883:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760781.002893:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28102656, 28106752) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.002936:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28102656=0x1acd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.002983:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.002986:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28106752 00000020:00010000:2.0:1789760781.002994:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.002997:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.003000:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.003004:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.003007:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.003014:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.003020:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.003034:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760781.003207:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 28102656<>28106752 00000100:00100000:3.0:1789760781.003298:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760781.003313:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512741888:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760781.049037:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974180 gen 3 00000008:00010000:3.0:1789760781.049076:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760781.050588:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.050605:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.050695:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760781.050723:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512741888:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760781.050807:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6861, 6861] 00000080:00200000:2.0:1789760781.050820:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.050826:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.050838:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28106752 i_blocks 55303 00000020:00200000:2.0:1789760781.050842:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.050850:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.050876:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28106752, count: 4096 00000080:00200000:2.0:1789760781.050882:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28106752, bytes: 4096 00000080:00200000:2.0:1789760781.050891:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.050915:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.050918:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28106752, 28110848) 0 0 00000080:00200000:2.0:1789760781.050922:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.050937:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6862, 6862] 00020000:00200000:2.0:1789760781.050945:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28106752 chunk: [28106752, 28110848) 28110848, 4096 00020000:00200000:2.0:1789760781.050949:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.050964:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6862, 6862] 00000008:00010000:2.0:1789760781.050981:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.050986:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760781.050995:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28106752, 28110848) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.051005:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28106752=0x1ace000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.051046:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.051049:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28110848 00000020:00010000:2.0:1789760781.051055:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.051057:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.051060:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.051063:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.051066:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.051073:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.051079:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.051094:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760781.051446:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 28106752<>28110848 00000100:00100000:3.0:1789760781.051510:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512742016:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760781.085238:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760781.086005:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974181 gen 3 00000008:00010000:3.0:1789760781.086038:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760781.086057:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.086145:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.086162:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760781.086184:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512742016:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760781.086276:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6862, 6862] 00000080:00200000:2.0:1789760781.086288:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.086293:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.086304:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28110848 i_blocks 55303 00000020:00200000:2.0:1789760781.086308:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.086316:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.086343:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28110848, count: 4096 00000080:00200000:2.0:1789760781.086350:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28110848, bytes: 4096 00000080:00200000:2.0:1789760781.086357:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.086365:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.086367:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28110848, 28114944) 0 0 00000080:00200000:2.0:1789760781.086370:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.086386:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6863, 6863] 00020000:00200000:2.0:1789760781.086393:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28110848 chunk: [28110848, 28114944) 28114944, 4096 00020000:00200000:2.0:1789760781.086397:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.086414:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6863, 6863] 00000008:00010000:2.0:1789760781.086429:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.086434:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760781.086444:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28110848, 28114944) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.086455:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28110848=0x1acf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.086500:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.086503:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28114944 00000020:00010000:2.0:1789760781.086509:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.086513:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.086516:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.086521:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.086524:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.086541:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.086548:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.086561:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760781.086634:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 28110848<>28114944 00000100:00100000:3.0:1789760781.087197:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760781.087216:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512742144:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760781.130185:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974182 gen 3 00000008:00010000:3.0:1789760781.130224:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760781.130231:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.130239:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.130299:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760781.130322:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512742144:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760781.130619:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6863, 6863] 00000080:00200000:2.0:1789760781.130632:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.130637:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.130649:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28114944 i_blocks 55303 00000020:00200000:2.0:1789760781.130653:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.130660:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.130686:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28114944, count: 4096 00000080:00200000:2.0:1789760781.130692:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28114944, bytes: 4096 00000080:00200000:2.0:1789760781.130699:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.130706:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.130708:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28114944, 28119040) 0 0 00000080:00200000:2.0:1789760781.130712:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.130725:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6864, 6864] 00020000:00200000:2.0:1789760781.130731:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28114944 chunk: [28114944, 28119040) 28119040, 4096 00020000:00200000:2.0:1789760781.130735:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.130751:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6864, 6864] 00000008:00010000:2.0:1789760781.130766:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.130771:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760781.130779:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28114944, 28119040) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.130885:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28114944=0x1ad0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.130966:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.130969:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28119040 00000020:00010000:2.0:1789760781.130977:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.130980:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.130982:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.130986:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.130989:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.130995:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.131001:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.131017:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760781.131091:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 28114944<>28119040 00000100:00100000:3.0:1789760781.131156:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512742272:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760781.165537:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974183 gen 3 00000008:00010000:3.0:1789760781.165571:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760781.165577:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.165586:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.165604:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760781.165628:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512742272:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760781.165759:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6864, 6864] 00000080:00200000:2.0:1789760781.165773:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.165778:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.165856:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28119040 i_blocks 55303 00000020:00200000:2.0:1789760781.165860:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.165869:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.165897:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28119040, count: 4096 00000080:00200000:2.0:1789760781.166026:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28119040, bytes: 4096 00000080:00200000:2.0:1789760781.166034:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.166041:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.166043:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28119040, 28123136) 0 0 00000080:00200000:2.0:1789760781.166047:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.166060:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6865, 6865] 00020000:00200000:2.0:1789760781.166068:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28119040 chunk: [28119040, 28123136) 28123136, 4096 00020000:00200000:2.0:1789760781.166072:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.166089:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6865, 6865] 00000008:00010000:2.0:1789760781.166104:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.166109:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760781.166118:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28119040, 28123136) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.166138:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28119040=0x1ad1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.166182:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.166185:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28123136 00000020:00010000:2.0:1789760781.166192:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.166194:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.166196:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.166199:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.166201:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.166206:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.166211:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.166221:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760781.166292:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 28119040<>28123136 00000100:00100000:3.0:1789760781.166378:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760781.166392:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512742400:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760781.195788:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760781.195799:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.195810:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.195867:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760781.195891:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512742400:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760781.196351:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6865, 6865] 00000080:00200000:2.0:1789760781.196363:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.196368:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.196380:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28123136 i_blocks 55303 00000020:00200000:2.0:1789760781.196383:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.196390:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.196414:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28123136, count: 4096 00000080:00200000:2.0:1789760781.196421:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28123136, bytes: 4096 00000080:00200000:2.0:1789760781.196428:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.196435:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.196437:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28123136, 28127232) 0 0 00000080:00200000:2.0:1789760781.196441:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.196608:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6866, 6866] 00020000:00200000:2.0:1789760781.196615:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28123136 chunk: [28123136, 28127232) 28127232, 4096 00020000:00200000:2.0:1789760781.196618:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.196632:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6866, 6866] 00000008:00010000:2.0:1789760781.196645:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.196649:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760781.196657:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28123136, 28127232) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.196666:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28123136=0x1ad2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.196709:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.196712:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28127232 00000020:00010000:2.0:1789760781.196718:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.196721:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.196724:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.196727:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.196730:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.196736:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.196742:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.196756:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760781.196824:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 28123136<>28127232 00000100:00100000:3.0:1789760781.196884:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512742528:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760781.235133:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974185 gen 3 00000008:00010000:3.0:1789760781.235180:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760781.235186:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.235194:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.235210:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760781.235241:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512742528:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760781.235427:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6866, 6866] 00000080:00200000:2.0:1789760781.235441:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.235446:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.235457:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28127232 i_blocks 55303 00000020:00200000:2.0:1789760781.235461:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.235469:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.235493:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28127232, count: 4096 00000080:00200000:2.0:1789760781.235500:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28127232, bytes: 4096 00000080:00200000:2.0:1789760781.235507:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.235515:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.235517:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28127232, 28131328) 0 0 00000080:00200000:2.0:1789760781.235520:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.235532:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6867, 6867] 00020000:00200000:2.0:1789760781.235538:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28127232 chunk: [28127232, 28131328) 28131328, 4096 00020000:00200000:2.0:1789760781.235541:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.235554:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6867, 6867] 00000008:00010000:2.0:1789760781.235568:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.235572:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760781.235581:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28127232, 28131328) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.235591:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28127232=0x1ad3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.235635:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.235638:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28131328 00000020:00010000:2.0:1789760781.235645:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.235649:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.235651:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.235655:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.235658:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.235665:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.235670:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.235684:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760781.235759:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 28127232<>28131328 00000100:00100000:3.0:1789760781.235816:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512742656:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760781.272803:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974186 gen 3 00000008:00010000:3.0:1789760781.272840:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760781.272846:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.272855:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.272872:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760781.272897:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512742656:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760781.273314:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6867, 6867] 00000080:00200000:2.0:1789760781.273328:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.273333:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.273345:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28131328 i_blocks 55303 00000020:00200000:2.0:1789760781.273348:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.273356:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.274140:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28131328, count: 4096 00000080:00200000:2.0:1789760781.274149:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28131328, bytes: 4096 00000080:00200000:2.0:1789760781.274158:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.274166:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.274168:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28131328, 28135424) 0 0 00000080:00200000:2.0:1789760781.274172:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.274186:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6868, 6868] 00020000:00200000:2.0:1789760781.274193:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28131328 chunk: [28131328, 28135424) 28135424, 4096 00020000:00200000:2.0:1789760781.274197:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.274212:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6868, 6868] 00000008:00010000:2.0:1789760781.274299:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.274304:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760781.274314:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28131328, 28135424) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.274326:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28131328=0x1ad4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.274372:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.274375:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28135424 00000020:00010000:2.0:1789760781.274383:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.274386:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.274389:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.274392:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.274394:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.274401:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.274407:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.274421:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760781.274499:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 28131328<>28135424 00000100:00100000:3.0:1789760781.274686:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512742784:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760781.330391:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974187 gen 3 00000008:00010000:3.0:1789760781.330430:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760781.330435:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.330443:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.330461:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760781.330483:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512742784:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760781.330863:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6868, 6868] 00000080:00200000:2.0:1789760781.330876:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.330880:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.330891:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28135424 i_blocks 55303 00000020:00200000:2.0:1789760781.330895:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.330938:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.330964:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28135424, count: 4096 00000080:00200000:2.0:1789760781.330971:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28135424, bytes: 4096 00000080:00200000:2.0:1789760781.330979:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.330986:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.330989:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28135424, 28139520) 0 0 00000080:00200000:2.0:1789760781.330993:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.331009:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6869, 6869] 00020000:00200000:2.0:1789760781.331016:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28135424 chunk: [28135424, 28139520) 28139520, 4096 00020000:00200000:2.0:1789760781.331020:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.331035:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6869, 6869] 00000008:00010000:2.0:1789760781.331048:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.331053:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760781.331062:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28135424, 28139520) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.331072:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28135424=0x1ad5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.331119:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.331121:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28139520 00000020:00010000:2.0:1789760781.331137:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.331141:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.331143:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.331147:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.331150:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.331156:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.331162:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.331177:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760781.331322:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 28135424<>28139520 00000100:00100000:3.0:1789760781.331482:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760781.331501:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512742912:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760781.380895:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974188 gen 3 00000008:00010000:3.0:1789760781.381076:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760781.381083:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.381092:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.381288:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760781.381314:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512742912:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760781.381381:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6869, 6869] 00000080:00200000:2.0:1789760781.381392:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.381397:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.381408:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28139520 i_blocks 55303 00000020:00200000:2.0:1789760781.381411:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.381419:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.381444:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28139520, count: 4096 00000080:00200000:2.0:1789760781.381450:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28139520, bytes: 4096 00000080:00200000:2.0:1789760781.381456:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.381464:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.381466:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28139520, 28143616) 0 0 00000080:00200000:2.0:1789760781.381469:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.381481:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6870, 6870] 00020000:00200000:2.0:1789760781.381488:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28139520 chunk: [28139520, 28143616) 28143616, 4096 00020000:00200000:2.0:1789760781.381491:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.381505:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6870, 6870] 00000008:00010000:2.0:1789760781.381518:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.381522:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760781.381532:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28139520, 28143616) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.381542:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28139520=0x1ad6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.381584:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.381586:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28143616 00000020:00010000:2.0:1789760781.381592:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.381595:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.381598:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.381601:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.381604:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.381610:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.381615:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.381628:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760781.381702:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 28139520<>28143616 00000100:00100000:3.0:1789760781.381816:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512743040:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760781.423089:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974189 gen 3 00000008:00010000:3.0:1789760781.423141:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760781.423148:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.423156:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.423207:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760781.423230:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512743040:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760781.423322:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6870, 6870] 00000080:00200000:2.0:1789760781.423335:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.423340:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.423351:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28143616 i_blocks 55303 00000020:00200000:2.0:1789760781.423355:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.423362:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.423389:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28143616, count: 4096 00000080:00200000:2.0:1789760781.423395:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28143616, bytes: 4096 00000080:00200000:2.0:1789760781.423475:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.423483:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.423485:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28143616, 28147712) 0 0 00000080:00200000:2.0:1789760781.423489:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.423502:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6871, 6871] 00020000:00200000:2.0:1789760781.423509:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28143616 chunk: [28143616, 28147712) 28147712, 4096 00020000:00200000:2.0:1789760781.423512:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.423527:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6871, 6871] 00000008:00010000:2.0:1789760781.423539:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.423544:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760781.423553:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28143616, 28147712) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.423564:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28143616=0x1ad7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.423737:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.423741:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28147712 00000020:00010000:2.0:1789760781.423749:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.423753:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.423756:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.423761:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.423764:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.423770:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.423776:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.423790:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760781.423867:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 28143616<>28147712 00000100:00100000:3.0:1789760781.423927:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760781.423944:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512743168:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760781.457596:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974190 gen 3 00000008:00010000:3.0:1789760781.457632:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760781.457639:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.457646:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.457666:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760781.457689:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512743168:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760781.457995:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6871, 6871] 00000080:00200000:2.0:1789760781.458007:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.458012:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.458024:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28147712 i_blocks 55303 00000020:00200000:2.0:1789760781.458027:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.458034:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.458058:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28147712, count: 4096 00000080:00200000:2.0:1789760781.458064:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28147712, bytes: 4096 00000080:00200000:2.0:1789760781.458072:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.458078:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.458081:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28147712, 28151808) 0 0 00000080:00200000:2.0:1789760781.458085:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.458097:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6872, 6872] 00020000:00200000:2.0:1789760781.458267:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28147712 chunk: [28147712, 28151808) 28151808, 4096 00020000:00200000:2.0:1789760781.458272:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.458287:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6872, 6872] 00000008:00010000:2.0:1789760781.458299:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.458304:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760781.458312:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28147712, 28151808) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.458323:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28147712=0x1ad8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.458364:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.458367:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28151808 00000020:00010000:2.0:1789760781.458373:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.458376:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.458379:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.458384:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.458386:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.458392:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.458398:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.458412:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760781.458478:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 28147712<>28151808 00000100:00100000:2.0:1789760781.458514:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760781.458527:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512743296:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760781.503572:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760781.503583:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.503592:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.504055:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760781.504083:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512743296:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760781.506554:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6872, 6872] 00000080:00200000:2.0:1789760781.506576:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.506582:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.506598:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28151808 i_blocks 55303 00000020:00200000:2.0:1789760781.506603:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.506615:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.506643:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28151808, count: 4096 00000080:00200000:2.0:1789760781.506650:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28151808, bytes: 4096 00000080:00200000:2.0:1789760781.506661:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.506668:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.506670:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28151808, 28155904) 0 0 00000080:00200000:2.0:1789760781.506673:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.506686:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6873, 6873] 00020000:00200000:2.0:1789760781.506693:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28151808 chunk: [28151808, 28155904) 28155904, 4096 00020000:00200000:2.0:1789760781.506697:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.506712:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6873, 6873] 00000008:00010000:2.0:1789760781.506726:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.506731:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760781.506741:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28151808, 28155904) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.506752:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28151808=0x1ad9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.506807:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.506809:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28155904 00000020:00010000:2.0:1789760781.506816:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.506820:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.506823:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.506826:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.506828:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.506835:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.506854:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.506876:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760781.506976:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 28151808<>28155904 00000100:00100000:3.0:1789760781.507215:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760781.507234:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512743424:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760781.545652:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974191 gen 3 00000100:00100000:3.0:1789760781.545681:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368f480 x1876699512743424/t4294974192(4294974192) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760797 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760781.545716:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760781.545721:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.545728:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.545783:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760781.545819:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512743424:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760781.546004:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6873, 6873] 00000080:00200000:2.0:1789760781.546019:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.546024:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.546037:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28155904 i_blocks 55303 00000020:00200000:2.0:1789760781.546041:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.546049:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.546074:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28155904, count: 4096 00000080:00200000:2.0:1789760781.546080:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28155904, bytes: 4096 00000080:00200000:2.0:1789760781.546088:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.546095:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.546097:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28155904, 28160000) 0 0 00000080:00200000:2.0:1789760781.546100:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.546113:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6874, 6874] 00020000:00200000:2.0:1789760781.546120:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28155904 chunk: [28155904, 28160000) 28160000, 4096 00020000:00200000:2.0:1789760781.546135:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.546150:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6874, 6874] 00000008:00010000:2.0:1789760781.546166:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.546170:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760781.546179:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28155904, 28160000) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.546259:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28155904=0x1ada000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.546304:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.546307:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28160000 00000020:00010000:2.0:1789760781.546314:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.546318:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.546321:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.546324:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.546326:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.546332:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.546338:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.546351:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760781.546426:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 28155904<>28160000 00000100:00100000:3.0:1789760781.546487:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512743552:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760781.590669:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974193 gen 3 00000008:00010000:3.0:1789760781.590723:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760781.590729:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.590737:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.590756:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760781.590781:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512743552:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760781.590795:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6874, 6874] 00000080:00200000:2.0:1789760781.590809:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.590815:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.590826:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28160000 i_blocks 55303 00000020:00200000:2.0:1789760781.590829:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.590837:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.590862:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28160000, count: 4096 00000080:00200000:2.0:1789760781.590868:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28160000, bytes: 4096 00000080:00200000:2.0:1789760781.590876:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.590883:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.590885:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28160000, 28164096) 0 0 00000080:00200000:2.0:1789760781.590888:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.590938:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6875, 6875] 00020000:00200000:2.0:1789760781.590945:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28160000 chunk: [28160000, 28164096) 28164096, 4096 00020000:00200000:2.0:1789760781.590949:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.590963:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6875, 6875] 00000008:00010000:2.0:1789760781.590978:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.590983:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760781.590992:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28160000, 28164096) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.591002:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28160000=0x1adb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.591044:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.591047:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28164096 00000020:00010000:2.0:1789760781.591053:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.591057:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.591059:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.591062:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.591065:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.591071:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.591077:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.591090:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760781.591173:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 28160000<>28164096 00000100:00100000:3.0:1789760781.591315:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760781.591329:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512743680:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760781.646507:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974194 gen 3 00000008:00010000:3.0:1789760781.646612:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760781.646619:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.646627:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.646712:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760781.646739:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512743680:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760781.646930:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6875, 6875] 00000080:00200000:0.0:1789760781.646943:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.646948:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.646961:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28164096 i_blocks 55303 00000020:00200000:0.0:1789760781.646964:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.646972:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.647090:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28164096, count: 4096 00000080:00200000:0.0:1789760781.647098:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28164096, bytes: 4096 00000080:00200000:0.0:1789760781.647106:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.647113:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.647115:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28164096, 28168192) 0 0 00000080:00200000:0.0:1789760781.647120:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.647144:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6876, 6876] 00020000:00200000:0.0:1789760781.647152:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28164096 chunk: [28164096, 28168192) 28168192, 4096 00020000:00200000:0.0:1789760781.647157:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.647176:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6876, 6876] 00000008:00010000:0.0:1789760781.647190:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.647196:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760781.647206:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28164096, 28168192) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.647386:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28164096=0x1adc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.647442:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.647445:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28168192 00000020:00010000:0.0:1789760781.647452:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.647456:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.647459:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.647463:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.647466:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.647474:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.647480:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.647498:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc300 00000008:00100000:0.0:1789760781.647651:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149c00 - object 0x240000400:5 offset 28164096<>28168192 00000100:00100000:1.0:1789760781.647734:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512743808:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760781.704224:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760781.704236:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.704246:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.704347:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760781.704374:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512743808:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760781.704437:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6876, 6876] 00000080:00200000:2.0:1789760781.704449:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.704455:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.704465:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28168192 i_blocks 55303 00000020:00200000:2.0:1789760781.704469:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.704476:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.704501:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28168192, count: 4096 00000080:00200000:2.0:1789760781.704507:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28168192, bytes: 4096 00000080:00200000:2.0:1789760781.704515:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.704522:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.704525:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28168192, 28172288) 0 0 00000080:00200000:2.0:1789760781.704529:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.704543:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6877, 6877] 00020000:00200000:2.0:1789760781.704550:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28168192 chunk: [28168192, 28172288) 28172288, 4096 00020000:00200000:2.0:1789760781.704554:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.704572:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6877, 6877] 00000008:00010000:2.0:1789760781.704586:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.704591:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760781.704601:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28168192, 28172288) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.704612:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28168192=0x1add000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.704656:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.704658:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28172288 00000020:00010000:2.0:1789760781.704665:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.704669:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.704672:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.704675:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.704678:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.704685:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.704690:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.704703:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760781.704968:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 28168192<>28172288 00000100:00100000:3.0:1789760781.705040:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512743936:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760781.739522:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974196 gen 3 00000008:00010000:3.0:1789760781.739574:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760781.739582:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760781.739613:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760781.739634:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760781.739663:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512743936:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760781.740055:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6877, 6877] 00000080:00200000:2.0:1789760781.740072:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760781.740077:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760781.740090:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28172288 i_blocks 55303 00000020:00200000:2.0:1789760781.740094:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760781.740104:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760781.740141:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28172288, count: 4096 00000080:00200000:2.0:1789760781.740504:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28172288, bytes: 4096 00000080:00200000:2.0:1789760781.740514:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760781.740523:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760781.740525:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28172288, 28176384) 0 0 00000080:00200000:2.0:1789760781.740529:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760781.740542:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6878, 6878] 00020000:00200000:2.0:1789760781.740550:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28172288 chunk: [28172288, 28176384) 28176384, 4096 00020000:00200000:2.0:1789760781.740553:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760781.740570:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6878, 6878] 00000008:00010000:2.0:1789760781.740584:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760781.740589:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760781.740598:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28172288, 28176384) append 0 append lockpos 0 00000080:00200000:2.0:1789760781.740609:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28172288=0x1ade000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760781.740657:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760781.740660:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28176384 00000020:00010000:2.0:1789760781.740666:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760781.740669:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.740672:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760781.740675:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760781.740678:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760781.740684:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760781.740689:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760781.740705:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760781.740785:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 28172288<>28176384 00000100:00100000:3.0:1789760781.740935:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760781.740952:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512744064:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760781.778463:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974197 gen 3 00000008:00010000:2.0:1789760781.778816:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:2.0:1789760781.778826:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760781.778838:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760781.778943:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:2.0:1789760781.778981:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512744064:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760781.779093:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6878, 6878] 00000080:00200000:0.0:1789760781.779312:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.779321:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.779437:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28176384 i_blocks 55303 00000020:00200000:0.0:1789760781.779444:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.779461:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.779510:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28176384, count: 4096 00000080:00200000:0.0:1789760781.779585:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28176384, bytes: 4096 00000080:00200000:0.0:1789760781.779606:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.779617:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.779619:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28176384, 28180480) 0 0 00000080:00200000:0.0:1789760781.779624:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.779714:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6879, 6879] 00020000:00200000:0.0:1789760781.779726:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28176384 chunk: [28176384, 28180480) 28180480, 4096 00020000:00200000:0.0:1789760781.779732:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.779760:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6879, 6879] 00000008:00010000:0.0:1789760781.779781:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.779788:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760781.779801:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28176384, 28180480) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.779817:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28176384=0x1adf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.779891:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.779894:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28180480 00000020:00010000:0.0:1789760781.780053:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.780058:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.780061:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.780065:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.780068:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.780077:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.780085:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.780297:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc240 00000008:00100000:0.0:1789760781.780507:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148000 - object 0x240000400:5 offset 28176384<>28180480 00000100:00100000:1.0:1789760781.780663:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512744192:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760781.839069:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974198 gen 3 00000008:00010000:1.0:1789760781.839148:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760781.839159:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.839171:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.839304:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760781.839409:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512744192:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760781.839603:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6879, 6879] 00000080:00200000:0.0:1789760781.839621:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.839628:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.839646:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28180480 i_blocks 55303 00000020:00200000:0.0:1789760781.839650:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.839738:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.839773:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28180480, count: 4096 00000080:00200000:0.0:1789760781.839782:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28180480, bytes: 4096 00000080:00200000:0.0:1789760781.839794:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.839804:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.839806:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28180480, 28184576) 0 0 00000080:00200000:0.0:1789760781.839811:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.839829:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6880, 6880] 00020000:00200000:0.0:1789760781.839839:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28180480 chunk: [28180480, 28184576) 28184576, 4096 00020000:00200000:0.0:1789760781.839844:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.839864:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6880, 6880] 00000008:00010000:0.0:1789760781.839886:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.839892:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760781.839941:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28180480, 28184576) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.839957:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28180480=0x1ae0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.840598:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.840603:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28184576 00000020:00010000:0.0:1789760781.840613:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.840618:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.840621:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.840624:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.840628:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.840637:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.840645:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.843202:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc480 00000008:00100000:0.0:1789760781.843362:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149180 - object 0x240000400:5 offset 28180480<>28184576 00000100:00100000:1.0:1789760781.843731:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512744320:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760781.881477:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974199 gen 3 00000008:00010000:1.0:1789760781.881568:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760781.881575:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.881585:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.881744:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760781.881767:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512744320:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760781.881837:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6880, 6880] 00000080:00200000:0.0:1789760781.881850:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.881855:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.881866:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28184576 i_blocks 55303 00000020:00200000:0.0:1789760781.881870:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.881878:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.881924:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28184576, count: 4096 00000080:00200000:0.0:1789760781.881930:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28184576, bytes: 4096 00000080:00200000:0.0:1789760781.881938:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.881945:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.881947:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28184576, 28188672) 0 0 00000080:00200000:0.0:1789760781.881951:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.881964:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6881, 6881] 00020000:00200000:0.0:1789760781.881970:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28184576 chunk: [28184576, 28188672) 28188672, 4096 00020000:00200000:0.0:1789760781.881974:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.881987:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6881, 6881] 00000008:00010000:0.0:1789760781.882000:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.882005:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760781.882014:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28184576, 28188672) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.882024:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28184576=0x1ae1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.882063:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.882066:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28188672 00000020:00010000:0.0:1789760781.882072:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.882075:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.882078:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.882081:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.882084:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.882090:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.882095:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.882108:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc840 00000008:00100000:0.0:1789760781.882457:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b480 - object 0x240000400:5 offset 28184576<>28188672 00000100:00100000:1.0:1789760781.882527:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512744448:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760781.925689:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974200 gen 3 00000008:00010000:1.0:1789760781.925722:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760781.925729:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.925738:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.925757:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760781.925780:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512744448:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760781.925867:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6881, 6881] 00000080:00200000:0.0:1789760781.925881:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.925886:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.925897:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28188672 i_blocks 55303 00000020:00200000:0.0:1789760781.925919:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.925927:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.925952:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28188672, count: 4096 00000080:00200000:0.0:1789760781.925958:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28188672, bytes: 4096 00000080:00200000:0.0:1789760781.925965:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.925972:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.925974:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28188672, 28192768) 0 0 00000080:00200000:0.0:1789760781.925978:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.925991:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6882, 6882] 00020000:00200000:0.0:1789760781.925997:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28188672 chunk: [28188672, 28192768) 28192768, 4096 00020000:00200000:0.0:1789760781.926000:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.926014:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6882, 6882] 00000008:00010000:0.0:1789760781.926027:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.926032:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760781.926040:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28188672, 28192768) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.926050:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28188672=0x1ae2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.926090:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.926093:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28192768 00000020:00010000:0.0:1789760781.926100:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.926104:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.926106:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.926109:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.926112:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.926118:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.926136:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.926151:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc840 00000008:00100000:0.0:1789760781.926281:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b800 - object 0x240000400:5 offset 28188672<>28192768 00000100:00100000:1.0:1789760781.926495:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512744576:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760781.957393:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974201 gen 3 00000008:00010000:1.0:1789760781.957433:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760781.957440:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.957449:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.957469:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760781.957492:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512744576:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760781.957518:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6882, 6882] 00000080:00200000:0.0:1789760781.957532:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.957538:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.957549:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28192768 i_blocks 55303 00000020:00200000:0.0:1789760781.957554:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.957561:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.957587:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28192768, count: 4096 00000080:00200000:0.0:1789760781.957593:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28192768, bytes: 4096 00000080:00200000:0.0:1789760781.957600:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.957608:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.957610:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28192768, 28196864) 0 0 00000080:00200000:0.0:1789760781.957613:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.957626:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6883, 6883] 00020000:00200000:0.0:1789760781.957633:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28192768 chunk: [28192768, 28196864) 28196864, 4096 00020000:00200000:0.0:1789760781.957637:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.957651:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6883, 6883] 00000008:00010000:0.0:1789760781.957664:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.957669:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760781.957678:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28192768, 28196864) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.957688:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28192768=0x1ae3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.957729:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.957731:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28196864 00000020:00010000:0.0:1789760781.957739:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.957742:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.957744:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.957747:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.957750:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.957757:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.957762:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.957776:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc480 00000008:00100000:0.0:1789760781.957858:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148380 - object 0x240000400:5 offset 28192768<>28196864 00000100:00100000:1.0:1789760781.958023:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512744704:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760781.995384:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974202 gen 3 00000008:00010000:1.0:1789760781.995432:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760781.995439:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760781.995448:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760781.995475:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760781.995499:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512744704:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760781.995698:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6883, 6883] 00000080:00200000:0.0:1789760781.995711:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760781.995717:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760781.995729:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28196864 i_blocks 55303 00000020:00200000:0.0:1789760781.995732:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760781.995740:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760781.995764:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28196864, count: 4096 00000080:00200000:0.0:1789760781.995770:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28196864, bytes: 4096 00000080:00200000:0.0:1789760781.995780:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760781.995787:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760781.995788:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28196864, 28200960) 0 0 00000080:00200000:0.0:1789760781.995792:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760781.995804:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6884, 6884] 00020000:00200000:0.0:1789760781.995810:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28196864 chunk: [28196864, 28200960) 28200960, 4096 00020000:00200000:0.0:1789760781.995814:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760781.995827:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6884, 6884] 00000008:00010000:0.0:1789760781.995850:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760781.995855:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760781.995864:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28196864, 28200960) append 0 append lockpos 0 00000080:00200000:0.0:1789760781.995875:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28196864=0x1ae4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760781.995960:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760781.995963:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28200960 00000020:00010000:0.0:1789760781.995969:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760781.995972:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.995975:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760781.995979:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760781.995981:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760781.995987:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760781.995992:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760781.996399:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc240 00000008:00100000:0.0:1789760781.996471:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149880 - object 0x240000400:5 offset 28196864<>28200960 00000100:00100000:1.0:1789760781.996521:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512744832:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760782.036377:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974203 gen 3 00000008:00010000:1.0:1789760782.036411:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760782.036417:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.036426:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.036452:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760782.036473:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512744832:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760782.037201:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6884, 6884] 00000080:00200000:0.0:1789760782.037215:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.037220:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.037421:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28200960 i_blocks 55303 00000020:00200000:0.0:1789760782.037426:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.037433:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.037458:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28200960, count: 4096 00000080:00200000:0.0:1789760782.037463:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28200960, bytes: 4096 00000080:00200000:0.0:1789760782.037477:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.037483:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.037484:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28200960, 28205056) 0 0 00000080:00200000:0.0:1789760782.037487:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.037499:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6885, 6885] 00020000:00200000:0.0:1789760782.037505:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28200960 chunk: [28200960, 28205056) 28205056, 4096 00020000:00200000:0.0:1789760782.037507:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.037519:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6885, 6885] 00000008:00010000:0.0:1789760782.037533:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.037538:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760782.037546:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28200960, 28205056) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.037558:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28200960=0x1ae5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.037635:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.037638:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28205056 00000020:00010000:0.0:1789760782.037644:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.037648:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.037651:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.037654:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.037657:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.037663:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.037669:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.037685:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc300 00000008:00100000:0.0:1789760782.037769:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a300 - object 0x240000400:5 offset 28200960<>28205056 00000100:00100000:1.0:1789760782.037831:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760782.037849:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512744960:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760782.070873:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974204 gen 3 00000008:00010000:1.0:1789760782.071329:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760782.071337:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.071346:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.071475:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760782.071499:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512744960:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760782.071769:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6885, 6885] 00000080:00200000:0.0:1789760782.071782:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.071787:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.071799:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28205056 i_blocks 55303 00000020:00200000:0.0:1789760782.071803:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.071810:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.071835:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28205056, count: 4096 00000080:00200000:0.0:1789760782.071842:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28205056, bytes: 4096 00000080:00200000:0.0:1789760782.072268:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.072278:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.072280:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28205056, 28209152) 0 0 00000080:00200000:0.0:1789760782.072285:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.072338:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6886, 6886] 00020000:00200000:0.0:1789760782.072346:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28205056 chunk: [28205056, 28209152) 28209152, 4096 00020000:00200000:0.0:1789760782.072350:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.072365:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6886, 6886] 00000008:00010000:0.0:1789760782.072378:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.072646:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760782.072656:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28205056, 28209152) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.072668:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28205056=0x1ae6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.072720:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.072723:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28209152 00000020:00010000:0.0:1789760782.072730:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.072733:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.072736:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.072739:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.072742:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.072748:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.072754:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.072766:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccb40 00000008:00100000:0.0:1789760782.072839:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a680 - object 0x240000400:5 offset 28205056<>28209152 00000100:00100000:1.0:1789760782.072937:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512745088:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760782.097339:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974205 gen 3 00000008:00010000:1.0:1789760782.097395:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760782.097402:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.097413:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.097502:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760782.097539:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512745088:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760782.097796:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6886, 6886] 00000080:00200000:0.0:1789760782.097812:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.097818:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.097830:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28209152 i_blocks 55303 00000020:00200000:0.0:1789760782.097833:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.097842:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.097867:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28209152, count: 4096 00000080:00200000:0.0:1789760782.097873:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28209152, bytes: 4096 00000080:00200000:0.0:1789760782.097881:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.097888:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.097890:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28209152, 28213248) 0 0 00000080:00200000:0.0:1789760782.097894:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.097932:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6887, 6887] 00020000:00200000:0.0:1789760782.097939:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28209152 chunk: [28209152, 28213248) 28213248, 4096 00020000:00200000:0.0:1789760782.097943:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.097958:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6887, 6887] 00000008:00010000:0.0:1789760782.097973:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.097978:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760782.097988:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28209152, 28213248) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.097999:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28209152=0x1ae7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.098043:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.098045:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28213248 00000020:00010000:0.0:1789760782.098052:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.098055:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.098058:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.098061:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.098063:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.098069:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.098074:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.098090:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000008:00100000:0.0:1789760782.098345:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148e00 - object 0x240000400:5 offset 28209152<>28213248 00000100:00100000:1.0:1789760782.098474:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512745216:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760782.109190:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760782.129530:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974206 gen 3 00000008:00010000:1.0:1789760782.129567:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760782.129573:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.129582:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.129594:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760782.129616:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512745216:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760782.129750:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6887, 6887] 00000080:00200000:3.0:1789760782.129763:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760782.129768:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760782.129779:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28213248 i_blocks 55303 00000020:00200000:3.0:1789760782.129783:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760782.129791:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760782.129817:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28213248, count: 4096 00000080:00200000:3.0:1789760782.129823:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28213248, bytes: 4096 00000080:00200000:3.0:1789760782.129830:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760782.129837:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760782.129839:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28213248, 28217344) 0 0 00000080:00200000:3.0:1789760782.129842:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760782.129855:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6888, 6888] 00020000:00200000:3.0:1789760782.129861:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28213248 chunk: [28213248, 28217344) 28217344, 4096 00020000:00200000:3.0:1789760782.129865:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760782.129878:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6888, 6888] 00000008:00010000:3.0:1789760782.129891:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760782.129897:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760782.129932:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28213248, 28217344) append 0 append lockpos 0 00000080:00200000:3.0:1789760782.129942:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28213248=0x1ae8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760782.129983:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760782.129986:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28217344 00000020:00010000:3.0:1789760782.129992:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760782.129995:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.129998:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.130001:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760782.130004:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760782.130010:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760782.130016:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760782.130030:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760782.130106:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac380 - object 0x240000400:5 offset 28213248<>28217344 00000100:00100000:3.0:1789760782.130156:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760782.130170:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512745344:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760782.171087:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974207 gen 3 00000008:00010000:3.0:1789760782.171140:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:3.0:1789760782.171147:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.171157:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.171182:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:3.0:1789760782.171406:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512745344:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760782.171565:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6888, 6888] 00000080:00200000:2.0:1789760782.171578:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760782.171582:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760782.171594:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28217344 i_blocks 55303 00000020:00200000:2.0:1789760782.171598:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760782.171606:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760782.171643:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28217344, count: 4096 00000080:00200000:2.0:1789760782.171649:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28217344, bytes: 4096 00000080:00200000:2.0:1789760782.171658:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760782.171665:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760782.171667:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28217344, 28221440) 0 0 00000080:00200000:2.0:1789760782.171671:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760782.171854:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6889, 6889] 00020000:00200000:2.0:1789760782.171862:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28217344 chunk: [28217344, 28221440) 28221440, 4096 00020000:00200000:2.0:1789760782.171865:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760782.171880:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6889, 6889] 00000008:00010000:2.0:1789760782.171893:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760782.171898:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760782.171955:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28217344, 28221440) append 0 append lockpos 0 00000080:00200000:2.0:1789760782.171966:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28217344=0x1ae9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760782.172003:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760782.172006:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28221440 00000020:00010000:2.0:1789760782.172012:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760782.172026:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760782.172029:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760782.172032:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760782.172035:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760782.172041:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760782.172046:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760782.172056:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760782.172141:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 28217344<>28221440 00000100:00100000:2.0:1789760782.172380:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760782.172395:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512745472:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760782.200492:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974208 gen 3 00000008:00010000:3.0:1789760782.200560:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760782.200567:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.200574:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.200592:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760782.200615:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512745472:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760782.200694:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6889, 6889] 00000080:00200000:2.0:1789760782.200704:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760782.200708:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760782.200717:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28221440 i_blocks 55303 00000020:00200000:2.0:1789760782.200720:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760782.200726:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760782.200746:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28221440, count: 4096 00000080:00200000:2.0:1789760782.200751:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28221440, bytes: 4096 00000080:00200000:2.0:1789760782.200756:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760782.200761:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760782.200763:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28221440, 28225536) 0 0 00000080:00200000:2.0:1789760782.200766:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760782.200776:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6890, 6890] 00020000:00200000:2.0:1789760782.200781:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28221440 chunk: [28221440, 28225536) 28225536, 4096 00020000:00200000:2.0:1789760782.200784:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760782.200796:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6890, 6890] 00000008:00010000:2.0:1789760782.200809:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760782.200815:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760782.200824:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28221440, 28225536) append 0 append lockpos 0 00000080:00200000:2.0:1789760782.200834:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28221440=0x1aea000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760782.200875:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760782.200878:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28225536 00000020:00010000:2.0:1789760782.200885:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760782.200888:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760782.200891:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760782.200894:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760782.200896:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760782.200930:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760782.200935:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760782.200945:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760782.201004:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 28221440<>28225536 00000100:00100000:2.0:1789760782.201022:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760782.201031:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512745600:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760782.243277:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974209 gen 3 00000008:00010000:3.0:1789760782.243375:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760782.243383:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760782.243392:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760782.243415:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760782.243441:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512745600:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760782.243587:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6890, 6890] 00000080:00200000:0.0:1789760782.243600:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.243605:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.243617:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28225536 i_blocks 55303 00000020:00200000:0.0:1789760782.243621:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.243629:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.243656:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28225536, count: 4096 00000080:00200000:0.0:1789760782.243663:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28225536, bytes: 4096 00000080:00200000:0.0:1789760782.243670:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.243691:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.243693:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28225536, 28229632) 0 0 00000080:00200000:0.0:1789760782.243696:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.243709:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6891, 6891] 00020000:00200000:0.0:1789760782.243715:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28225536 chunk: [28225536, 28229632) 28229632, 4096 00020000:00200000:0.0:1789760782.243719:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.243734:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6891, 6891] 00000008:00010000:0.0:1789760782.243747:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.243751:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760782.243760:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28225536, 28229632) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.243770:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28225536=0x1aeb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.243941:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.243945:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28229632 00000020:00010000:0.0:1789760782.243952:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.243956:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.243959:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.243962:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.243965:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.243971:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.243977:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.243991:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc900 00000008:00100000:0.0:1789760782.244056:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148a80 - object 0x240000400:5 offset 28225536<>28229632 00000100:00100000:1.0:1789760782.244479:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760782.244495:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512745728:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760782.284738:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760782.284750:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.284759:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.284781:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760782.284804:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512745728:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760782.285292:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6891, 6891] 00000080:00200000:0.0:1789760782.285305:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.285310:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.285321:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28229632 i_blocks 55303 00000020:00200000:0.0:1789760782.285325:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.285333:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.285356:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28229632, count: 4096 00000080:00200000:0.0:1789760782.285362:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28229632, bytes: 4096 00000080:00200000:0.0:1789760782.285370:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.285378:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.285381:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28229632, 28233728) 0 0 00000080:00200000:0.0:1789760782.285385:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.285401:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6892, 6892] 00020000:00200000:0.0:1789760782.285408:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28229632 chunk: [28229632, 28233728) 28233728, 4096 00020000:00200000:0.0:1789760782.285412:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.285428:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6892, 6892] 00000008:00010000:0.0:1789760782.285441:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.285445:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760782.285454:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28229632, 28233728) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.285465:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28229632=0x1aec000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.285511:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.285513:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28233728 00000020:00010000:0.0:1789760782.285520:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.285523:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.285526:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.285529:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.285532:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.285538:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.285543:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.285559:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cccc0 00000008:00100000:0.0:1789760782.285637:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514ad80 - object 0x240000400:5 offset 28229632<>28233728 00000100:00100000:0.0:1789760782.285686:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760782.285699:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512745856:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760782.324702:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974211 gen 3 00000008:00010000:0.0:1789760782.324750:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:0.0:1789760782.324757:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760782.324764:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760782.324824:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:0.0:1789760782.324847:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512745856:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760782.324933:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6892, 6892] 00000080:00200000:3.0:1789760782.324946:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760782.324951:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760782.324962:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28233728 i_blocks 55303 00000020:00200000:3.0:1789760782.324965:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760782.324973:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760782.324998:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28233728, count: 4096 00000080:00200000:3.0:1789760782.325004:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28233728, bytes: 4096 00000080:00200000:3.0:1789760782.325012:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760782.325019:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760782.325021:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28233728, 28237824) 0 0 00000080:00200000:3.0:1789760782.325025:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760782.325038:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6893, 6893] 00020000:00200000:3.0:1789760782.325045:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28233728 chunk: [28233728, 28237824) 28237824, 4096 00020000:00200000:3.0:1789760782.325049:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760782.325351:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6893, 6893] 00000008:00010000:3.0:1789760782.325366:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760782.325370:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760782.325380:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28233728, 28237824) append 0 append lockpos 0 00000080:00200000:3.0:1789760782.325391:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28233728=0x1aed000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760782.325438:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760782.325440:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28237824 00000020:00010000:3.0:1789760782.325449:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760782.325452:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.325455:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.325457:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760782.325459:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760782.325478:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760782.325484:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760782.325500:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760782.325936:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac380 - object 0x240000400:5 offset 28233728<>28237824 00000100:00100000:3.0:1789760782.325983:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760782.325999:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512745984:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760782.377679:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974212 gen 3 00000008:00010000:2.0:1789760782.377717:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760782.377724:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760782.377779:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760782.377835:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760782.377860:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512745984:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760782.377985:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6893, 6893] 00000080:00200000:3.0:1789760782.377998:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760782.378003:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760782.378016:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28237824 i_blocks 55303 00000020:00200000:3.0:1789760782.378019:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760782.378028:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760782.378057:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28237824, count: 4096 00000080:00200000:3.0:1789760782.378064:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28237824, bytes: 4096 00000080:00200000:3.0:1789760782.378696:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760782.378706:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760782.378708:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28237824, 28241920) 0 0 00000080:00200000:3.0:1789760782.378713:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760782.378728:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6894, 6894] 00020000:00200000:3.0:1789760782.378736:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28237824 chunk: [28237824, 28241920) 28241920, 4096 00020000:00200000:3.0:1789760782.378740:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760782.378758:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6894, 6894] 00000008:00010000:3.0:1789760782.378773:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760782.378778:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760782.378786:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28237824, 28241920) append 0 append lockpos 0 00000080:00200000:3.0:1789760782.378798:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28237824=0x1aee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760782.378849:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760782.378852:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28241920 00000020:00010000:3.0:1789760782.378858:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760782.378862:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.378864:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.378868:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760782.378871:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760782.378878:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760782.378884:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760782.379038:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760782.379403:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad180 - object 0x240000400:5 offset 28237824<>28241920 00000100:00100000:2.0:1789760782.379472:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512746112:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760782.423455:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760782.423469:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760782.423480:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760782.423541:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760782.423565:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512746112:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760782.423633:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6894, 6894] 00000080:00200000:3.0:1789760782.423645:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760782.423650:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760782.423661:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28241920 i_blocks 55303 00000020:00200000:3.0:1789760782.423664:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760782.423672:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760782.423700:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28241920, count: 4096 00000080:00200000:3.0:1789760782.423705:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28241920, bytes: 4096 00000080:00200000:3.0:1789760782.423713:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760782.423720:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760782.423722:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28241920, 28246016) 0 0 00000080:00200000:3.0:1789760782.423726:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760782.423740:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6895, 6895] 00020000:00200000:3.0:1789760782.423746:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28241920 chunk: [28241920, 28246016) 28246016, 4096 00020000:00200000:3.0:1789760782.423750:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760782.423766:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6895, 6895] 00000008:00010000:3.0:1789760782.423779:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760782.423784:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760782.423792:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28241920, 28246016) append 0 append lockpos 0 00000080:00200000:3.0:1789760782.423803:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28241920=0x1aef000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760782.423849:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760782.423852:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28246016 00000020:00010000:3.0:1789760782.423859:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760782.423863:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.423866:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.423869:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760782.423871:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760782.423877:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760782.424408:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760782.424426:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760782.424568:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 28241920<>28246016 00000100:00100000:2.0:1789760782.424636:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512746240:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760782.475842:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974214 gen 3 00000008:00010000:2.0:1789760782.475933:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760782.475940:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760782.475948:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760782.475966:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:2.0:1789760782.475990:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512746240:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760782.476076:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6895, 6895] 00000080:00200000:3.0:1789760782.476088:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760782.476093:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760782.476104:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28246016 i_blocks 55303 00000020:00200000:3.0:1789760782.476107:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760782.476115:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760782.476160:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28246016, count: 4096 00000080:00200000:3.0:1789760782.476165:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28246016, bytes: 4096 00000080:00200000:3.0:1789760782.476173:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760782.476180:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760782.476182:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28246016, 28250112) 0 0 00000080:00200000:3.0:1789760782.476186:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760782.476209:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6896, 6896] 00020000:00200000:3.0:1789760782.476216:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28246016 chunk: [28246016, 28250112) 28250112, 4096 00020000:00200000:3.0:1789760782.476220:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760782.476234:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6896, 6896] 00000008:00010000:3.0:1789760782.476248:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760782.476253:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760782.476263:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28246016, 28250112) append 0 append lockpos 0 00000080:00200000:3.0:1789760782.476273:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28246016=0x1af0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760782.476316:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760782.476320:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28250112 00000020:00010000:3.0:1789760782.476327:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760782.476330:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.476333:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.476336:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760782.476339:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760782.476345:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760782.476350:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760782.476362:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760782.476432:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aea00 - object 0x240000400:5 offset 28246016<>28250112 00000100:00100000:2.0:1789760782.476741:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512746368:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760782.518311:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974215 gen 3 00000008:00010000:2.0:1789760782.518461:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760782.518467:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760782.518476:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760782.518529:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760782.518553:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512746368:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760782.518623:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6896, 6896] 00000080:00200000:3.0:1789760782.518635:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760782.518641:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760782.518653:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28250112 i_blocks 55303 00000020:00200000:3.0:1789760782.518656:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760782.518665:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760782.518693:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28250112, count: 4096 00000080:00200000:3.0:1789760782.518699:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28250112, bytes: 4096 00000080:00200000:3.0:1789760782.518707:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760782.518714:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760782.518717:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28250112, 28254208) 0 0 00000080:00200000:3.0:1789760782.518720:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760782.518733:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6897, 6897] 00020000:00200000:3.0:1789760782.518740:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28250112 chunk: [28250112, 28254208) 28254208, 4096 00020000:00200000:3.0:1789760782.518744:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760782.518760:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6897, 6897] 00000008:00010000:3.0:1789760782.518774:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760782.518779:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760782.518788:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28250112, 28254208) append 0 append lockpos 0 00000080:00200000:3.0:1789760782.518799:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28250112=0x1af1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760782.518838:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760782.518841:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28254208 00000020:00010000:3.0:1789760782.519085:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760782.519090:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.519294:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.519297:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760782.519300:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760782.519306:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760782.519312:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760782.519384:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760782.519453:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adf80 - object 0x240000400:5 offset 28250112<>28254208 00000100:00100000:2.0:1789760782.519510:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512746496:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760782.559298:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974216 gen 3 00000008:00010000:2.0:1789760782.559333:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760782.559340:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760782.559349:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760782.559413:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760782.559434:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512746496:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760782.559503:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6897, 6897] 00000080:00200000:3.0:1789760782.559573:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760782.559579:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760782.559591:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28254208 i_blocks 55303 00000020:00200000:3.0:1789760782.559595:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760782.559603:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760782.559630:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28254208, count: 4096 00000080:00200000:3.0:1789760782.559636:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28254208, bytes: 4096 00000080:00200000:3.0:1789760782.559645:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760782.559652:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760782.559654:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28254208, 28258304) 0 0 00000080:00200000:3.0:1789760782.559658:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760782.559671:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6898, 6898] 00020000:00200000:3.0:1789760782.559687:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28254208 chunk: [28254208, 28258304) 28258304, 4096 00020000:00200000:3.0:1789760782.559691:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760782.559706:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6898, 6898] 00000008:00010000:3.0:1789760782.559720:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760782.559725:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760782.559734:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28254208, 28258304) append 0 append lockpos 0 00000080:00200000:3.0:1789760782.559743:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28254208=0x1af2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760782.559784:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760782.559787:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28258304 00000020:00010000:3.0:1789760782.560218:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760782.560223:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.560226:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760782.560229:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760782.560232:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760782.560239:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760782.560245:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760782.560319:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760782.560399:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af100 - object 0x240000400:5 offset 28254208<>28258304 00000100:00100000:3.0:1789760782.560444:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512746624:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760782.623112:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974217 gen 3 00000008:00010000:2.0:1789760782.623162:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760782.623169:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760782.623178:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760782.623188:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:2.0:1789760782.623212:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512746624:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760782.623461:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6898, 6898] 00000080:00200000:0.0:1789760782.623473:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.623478:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.623490:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28258304 i_blocks 55303 00000020:00200000:0.0:1789760782.623493:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.623501:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.623544:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28258304, count: 4096 00000080:00200000:0.0:1789760782.623551:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28258304, bytes: 4096 00000080:00200000:0.0:1789760782.623558:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.623566:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.623568:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28258304, 28262400) 0 0 00000080:00200000:0.0:1789760782.623572:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.623586:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6899, 6899] 00020000:00200000:0.0:1789760782.623593:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28258304 chunk: [28258304, 28262400) 28262400, 4096 00020000:00200000:0.0:1789760782.623597:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.623613:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6899, 6899] 00000008:00010000:0.0:1789760782.623628:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.623632:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760782.623642:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28258304, 28262400) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.623653:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28258304=0x1af3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.623697:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.623700:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28262400 00000020:00010000:0.0:1789760782.623707:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.623710:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.623713:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.623717:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.623719:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.623725:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.623730:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.623744:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cccc0 00000008:00100000:0.0:1789760782.623820:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514ad80 - object 0x240000400:5 offset 28258304<>28262400 00000100:00100000:1.0:1789760782.624180:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760782.624197:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512746752:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760782.717099:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974218 gen 3 00000008:00010000:1.0:1789760782.717149:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760782.717157:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.717167:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.717193:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760782.717217:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512746752:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760782.718102:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6899, 6899] 00000080:00200000:0.0:1789760782.718126:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.718132:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.718145:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28262400 i_blocks 55303 00000020:00200000:0.0:1789760782.718149:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.718159:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.718185:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28262400, count: 4096 00000080:00200000:0.0:1789760782.718190:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28262400, bytes: 4096 00000080:00200000:0.0:1789760782.718201:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.718208:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.718210:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28262400, 28266496) 0 0 00000080:00200000:0.0:1789760782.718214:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.718228:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6900, 6900] 00020000:00200000:0.0:1789760782.718235:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28262400 chunk: [28262400, 28266496) 28266496, 4096 00020000:00200000:0.0:1789760782.718238:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.718255:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6900, 6900] 00000008:00010000:0.0:1789760782.718269:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.718273:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760782.718282:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28262400, 28266496) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.718294:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28262400=0x1af4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.718343:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.718345:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28266496 00000020:00010000:0.0:1789760782.718352:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.718355:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.718357:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.718361:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.718363:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.718370:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.718375:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.718390:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000008:00100000:0.0:1789760782.718466:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148a80 - object 0x240000400:5 offset 28262400<>28266496 00000100:00100000:1.0:1789760782.718535:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512746880:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760782.796967:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974219 gen 3 00000008:00010000:1.0:1789760782.797004:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760782.797010:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.797019:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.797033:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760782.797056:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512746880:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760782.798364:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6900, 6900] 00000080:00200000:0.0:1789760782.798380:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.798385:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.798398:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28266496 i_blocks 55303 00000020:00200000:0.0:1789760782.798402:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.798412:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.798435:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28266496, count: 4096 00000080:00200000:0.0:1789760782.798440:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28266496, bytes: 4096 00000080:00200000:0.0:1789760782.798450:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.798457:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.798459:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28266496, 28270592) 0 0 00000080:00200000:0.0:1789760782.798463:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.798475:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6901, 6901] 00020000:00200000:0.0:1789760782.798481:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28266496 chunk: [28266496, 28270592) 28270592, 4096 00020000:00200000:0.0:1789760782.798485:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.798499:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6901, 6901] 00000008:00010000:0.0:1789760782.798512:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.798517:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760782.798526:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28266496, 28270592) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.798536:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28266496=0x1af5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.798579:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.798582:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28270592 00000020:00010000:0.0:1789760782.798588:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.798591:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.798594:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.798597:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.798600:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.798606:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.798611:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.798624:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc600 00000008:00100000:0.0:1789760782.798699:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148700 - object 0x240000400:5 offset 28266496<>28270592 00000100:00100000:0.0:1789760782.798741:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760782.798756:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512747008:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760782.857477:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974220 gen 3 00000008:00010000:0.0:1789760782.857511:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:0.0:1789760782.857518:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760782.857527:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760782.857607:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:0.0:1789760782.857630:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512747008:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760782.861332:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6901, 6901] 00000080:00200000:1.0:1789760782.861352:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760782.861356:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760782.861368:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28270592 i_blocks 55303 00000020:00200000:1.0:1789760782.861372:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760782.861386:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760782.861411:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28270592, count: 4096 00000080:00200000:1.0:1789760782.861418:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28270592, bytes: 4096 00000080:00200000:1.0:1789760782.861428:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760782.861435:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760782.861437:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28270592, 28274688) 0 0 00000080:00200000:1.0:1789760782.861440:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760782.861453:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6902, 6902] 00020000:00200000:1.0:1789760782.861459:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28270592 chunk: [28270592, 28274688) 28274688, 4096 00020000:00200000:1.0:1789760782.861463:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760782.861479:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6902, 6902] 00000008:00010000:1.0:1789760782.861493:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760782.861498:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760782.861507:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28270592, 28274688) append 0 append lockpos 0 00000080:00200000:1.0:1789760782.861518:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28270592=0x1af6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760782.861567:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760782.861569:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28274688 00000020:00010000:1.0:1789760782.861575:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760782.861578:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760782.861581:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760782.861584:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760782.861587:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760782.861594:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760782.861599:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760782.861621:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760782.861696:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97480 - object 0x240000400:5 offset 28270592<>28274688 00000100:00100000:1.0:1789760782.861738:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760782.861755:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512747136:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760782.925725:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974221 gen 3 00000008:00010000:1.0:1789760782.925758:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760782.925764:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760782.925772:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760782.925790:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760782.925809:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512747136:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760782.925922:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6902, 6902] 00000080:00200000:0.0:1789760782.925934:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760782.925939:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760782.925950:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28274688 i_blocks 55303 00000020:00200000:0.0:1789760782.925953:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760782.925960:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760782.925984:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28274688, count: 4096 00000080:00200000:0.0:1789760782.925989:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28274688, bytes: 4096 00000080:00200000:0.0:1789760782.925997:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760782.926004:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760782.926006:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28274688, 28278784) 0 0 00000080:00200000:0.0:1789760782.926010:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760782.926023:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6903, 6903] 00020000:00200000:0.0:1789760782.926030:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28274688 chunk: [28274688, 28278784) 28278784, 4096 00020000:00200000:0.0:1789760782.926034:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760782.926049:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6903, 6903] 00000008:00010000:0.0:1789760782.926078:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760782.926083:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760782.926091:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28274688, 28278784) append 0 append lockpos 0 00000080:00200000:0.0:1789760782.926101:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28274688=0x1af7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760782.926557:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760782.926560:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28278784 00000020:00010000:0.0:1789760782.926567:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760782.926570:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.926574:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760782.926577:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760782.926579:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760782.926586:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760782.926591:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760782.926608:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc600 00000008:00100000:0.0:1789760782.926686:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148700 - object 0x240000400:5 offset 28274688<>28278784 00000100:00100000:0.0:1789760782.926725:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760782.926737:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512747264:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760782.962676:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:0.0:1789760782.962687:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760782.962695:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760782.962744:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:0.0:1789760782.962765:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512747264:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760782.962819:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6903, 6903] 00000080:00200000:2.0:1789760782.962830:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760782.962835:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760782.962846:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28278784 i_blocks 55303 00000020:00200000:2.0:1789760782.962850:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760782.962857:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760782.962879:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28278784, count: 4096 00000080:00200000:2.0:1789760782.962885:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28278784, bytes: 4096 00000080:00200000:2.0:1789760782.962891:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760782.962898:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760782.962920:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28278784, 28282880) 0 0 00000080:00200000:2.0:1789760782.962924:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760782.962937:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6904, 6904] 00020000:00200000:2.0:1789760782.962944:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28278784 chunk: [28278784, 28282880) 28282880, 4096 00020000:00200000:2.0:1789760782.962948:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760782.962962:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6904, 6904] 00000008:00010000:2.0:1789760782.962975:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760782.962979:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760782.962987:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28278784, 28282880) append 0 append lockpos 0 00000080:00200000:2.0:1789760782.962998:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28278784=0x1af8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760782.963036:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760782.963038:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28282880 00000020:00010000:2.0:1789760782.963045:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760782.963048:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760782.963099:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760782.963102:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760782.963105:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760782.963120:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760782.963126:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760782.963138:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760782.963210:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 28278784<>28282880 00000100:00100000:3.0:1789760782.963278:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512747392:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760783.007915:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974223 gen 3 00000008:00010000:3.0:1789760783.008743:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760783.008752:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.008762:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.008833:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760783.008859:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512747392:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760783.008959:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6904, 6904] 00000080:00200000:2.0:1789760783.008971:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.008977:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.008987:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28282880 i_blocks 55303 00000020:00200000:2.0:1789760783.008990:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.008998:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.009023:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28282880, count: 4096 00000080:00200000:2.0:1789760783.009029:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28282880, bytes: 4096 00000080:00200000:2.0:1789760783.009038:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.009045:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.009047:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28282880, 28286976) 0 0 00000080:00200000:2.0:1789760783.009051:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.009064:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6905, 6905] 00020000:00200000:2.0:1789760783.009070:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28282880 chunk: [28282880, 28286976) 28286976, 4096 00020000:00200000:2.0:1789760783.009073:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.009088:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6905, 6905] 00000008:00010000:2.0:1789760783.009101:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.009106:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760783.009126:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28282880, 28286976) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.009138:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28282880=0x1af9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.009471:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.009474:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28286976 00000020:00010000:2.0:1789760783.009482:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.009485:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.009489:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.009492:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.009494:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.009501:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.009578:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.009595:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760783.009743:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 28282880<>28286976 00000100:00100000:3.0:1789760783.009813:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512747520:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760783.059952:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760783.059963:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.059973:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.059990:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760783.060015:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512747520:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760783.060139:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6905, 6905] 00000080:00200000:2.0:1789760783.060152:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.060157:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.060169:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28286976 i_blocks 55303 00000020:00200000:2.0:1789760783.060173:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.060181:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.060209:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28286976, count: 4096 00000080:00200000:2.0:1789760783.060216:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28286976, bytes: 4096 00000080:00200000:2.0:1789760783.060224:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.060231:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.060233:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28286976, 28291072) 0 0 00000080:00200000:2.0:1789760783.060237:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.060251:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6906, 6906] 00020000:00200000:2.0:1789760783.060258:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28286976 chunk: [28286976, 28291072) 28291072, 4096 00020000:00200000:2.0:1789760783.060262:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.060277:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6906, 6906] 00000008:00010000:2.0:1789760783.060292:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.060297:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760783.060306:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28286976, 28291072) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.060316:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28286976=0x1afa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.060372:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.060375:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28291072 00000020:00010000:2.0:1789760783.060382:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.060385:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.060388:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.060391:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.060394:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.060402:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.060408:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.060423:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760783.060501:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 28286976<>28291072 00000100:00100000:3.0:1789760783.060750:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512747648:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760783.108663:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974225 gen 3 00000008:00010000:3.0:1789760783.108711:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760783.108718:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.108725:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.108749:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760783.108774:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512747648:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760783.108878:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6906, 6906] 00000080:00200000:0.0:1789760783.108892:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760783.108897:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760783.108971:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28291072 i_blocks 55303 00000020:00200000:0.0:1789760783.108975:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760783.108984:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760783.109011:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28291072, count: 4096 00000080:00200000:0.0:1789760783.109016:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28291072, bytes: 4096 00000080:00200000:0.0:1789760783.109025:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760783.109032:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760783.109034:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28291072, 28295168) 0 0 00000080:00200000:0.0:1789760783.109038:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760783.109050:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6907, 6907] 00020000:00200000:0.0:1789760783.109056:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28291072 chunk: [28291072, 28295168) 28295168, 4096 00020000:00200000:0.0:1789760783.109060:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760783.109074:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6907, 6907] 00000008:00010000:0.0:1789760783.109086:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760783.109091:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760783.109099:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28291072, 28295168) append 0 append lockpos 0 00000080:00200000:0.0:1789760783.109109:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28291072=0x1afb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760783.109158:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760783.109160:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28295168 00000020:00010000:0.0:1789760783.109166:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760783.109169:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.109172:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.109175:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760783.109178:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760783.109183:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760783.109188:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760783.109198:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc600 00000008:00100000:0.0:1789760783.109270:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514aa00 - object 0x240000400:5 offset 28291072<>28295168 00000100:00100000:1.0:1789760783.109451:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512747776:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760783.133119:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:1.0:1789760783.162026:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760783.162039:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760783.162050:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760783.162070:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760783.162092:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512747776:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760783.162194:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6907, 6907] 00000080:00200000:0.0:1789760783.162206:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760783.162211:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760783.162224:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28295168 i_blocks 55303 00000020:00200000:0.0:1789760783.162227:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760783.162235:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760783.162263:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28295168, count: 4096 00000080:00200000:0.0:1789760783.162269:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28295168, bytes: 4096 00000080:00200000:0.0:1789760783.162276:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760783.162283:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760783.162286:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28295168, 28299264) 0 0 00000080:00200000:0.0:1789760783.162289:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760783.162301:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6908, 6908] 00020000:00200000:0.0:1789760783.162308:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28295168 chunk: [28295168, 28299264) 28299264, 4096 00020000:00200000:0.0:1789760783.162312:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760783.162328:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6908, 6908] 00000008:00010000:0.0:1789760783.162342:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760783.162346:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760783.162356:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28295168, 28299264) append 0 append lockpos 0 00000080:00200000:0.0:1789760783.162365:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28295168=0x1afc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760783.162411:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760783.162414:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28299264 00000020:00010000:0.0:1789760783.162420:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760783.162423:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.162426:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.162429:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760783.162432:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760783.162438:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760783.162444:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760783.162457:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc540 00000008:00100000:0.0:1789760783.162533:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148700 - object 0x240000400:5 offset 28295168<>28299264 00000100:00100000:1.0:1789760783.162691:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760783.162703:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512747904:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760783.196397:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974226 gen 3 00000100:00100000:1.0:1789760783.196427:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005148700 x1876699512747904/t4294974227(4294974227) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760799 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760783.196464:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760783.196469:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760783.196477:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760783.196566:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760783.196591:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512747904:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760783.197734:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6908, 6908] 00000080:00200000:0.0:1789760783.197752:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760783.197757:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760783.197769:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28299264 i_blocks 55303 00000020:00200000:0.0:1789760783.197773:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760783.197936:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760783.197965:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28299264, count: 4096 00000080:00200000:0.0:1789760783.197971:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28299264, bytes: 4096 00000080:00200000:0.0:1789760783.197982:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760783.197989:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760783.197991:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28299264, 28303360) 0 0 00000080:00200000:0.0:1789760783.197996:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760783.198007:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6909, 6909] 00020000:00200000:0.0:1789760783.198014:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28299264 chunk: [28299264, 28303360) 28303360, 4096 00020000:00200000:0.0:1789760783.198018:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760783.198034:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6909, 6909] 00000008:00010000:0.0:1789760783.198047:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760783.198052:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760783.198062:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28299264, 28303360) append 0 append lockpos 0 00000080:00200000:0.0:1789760783.198073:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28299264=0x1afd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760783.198127:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760783.198130:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28303360 00000020:00010000:0.0:1789760783.198137:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760783.198140:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.198143:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.198146:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760783.198148:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760783.198155:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760783.198160:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760783.198177:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cca80 00000008:00100000:0.0:1789760783.198256:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148a80 - object 0x240000400:5 offset 28299264<>28303360 00000100:00100000:0.0:1789760783.198302:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760783.198319:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512748032:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760783.236105:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974228 gen 3 00000008:00010000:0.0:1789760783.236232:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:0.0:1789760783.236238:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760783.236247:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760783.236268:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:0.0:1789760783.236292:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512748032:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760783.236372:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6909, 6909] 00000080:00200000:2.0:1789760783.236385:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.236390:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.236401:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28303360 i_blocks 55303 00000020:00200000:2.0:1789760783.236404:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.236413:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.236439:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28303360, count: 4096 00000080:00200000:2.0:1789760783.236445:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28303360, bytes: 4096 00000080:00200000:2.0:1789760783.236453:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.236460:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.236462:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28303360, 28307456) 0 0 00000080:00200000:2.0:1789760783.236466:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.236478:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6910, 6910] 00020000:00200000:2.0:1789760783.236485:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28303360 chunk: [28303360, 28307456) 28307456, 4096 00020000:00200000:2.0:1789760783.236489:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.236504:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6910, 6910] 00000008:00010000:2.0:1789760783.236519:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.236522:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760783.236531:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28303360, 28307456) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.236542:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28303360=0x1afe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.236578:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.236579:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28307456 00000020:00010000:2.0:1789760783.236584:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.236587:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.236589:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.236591:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.236593:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.236597:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.236602:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.236611:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760783.236686:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 28303360<>28307456 00000100:00100000:3.0:1789760783.236972:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760783.236987:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512748160:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760783.271979:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974229 gen 3 00000008:00010000:3.0:1789760783.272015:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760783.272022:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.272031:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.272047:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760783.272072:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512748160:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760783.272208:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6910, 6910] 00000080:00200000:2.0:1789760783.272220:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.272225:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.272236:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28307456 i_blocks 55303 00000020:00200000:2.0:1789760783.272239:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.272246:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.272271:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28307456, count: 4096 00000080:00200000:2.0:1789760783.272278:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28307456, bytes: 4096 00000080:00200000:2.0:1789760783.272285:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.272292:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.272294:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28307456, 28311552) 0 0 00000080:00200000:2.0:1789760783.272298:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.272310:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6911, 6911] 00020000:00200000:2.0:1789760783.272316:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28307456 chunk: [28307456, 28311552) 28311552, 4096 00020000:00200000:2.0:1789760783.272320:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.272335:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6911, 6911] 00000008:00010000:2.0:1789760783.272349:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.272353:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760783.272361:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28307456, 28311552) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.272371:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28307456=0x1aff000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.272414:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.272416:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28311552 00000020:00010000:2.0:1789760783.272422:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.272425:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.272428:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.272432:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.272435:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.272441:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.272446:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.272460:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760783.272548:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 28307456<>28311552 00000100:00100000:3.0:1789760783.272698:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512748288:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760783.304536:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974230 gen 3 00000008:00010000:3.0:1789760783.304574:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760783.304580:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.304589:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.304608:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760783.304632:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512748288:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760783.304800:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6911, 6911] 00000080:00200000:2.0:1789760783.304814:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.304820:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.304832:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28311552 i_blocks 55303 00000020:00200000:2.0:1789760783.304836:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.304843:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.304868:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28311552, count: 4096 00000080:00200000:2.0:1789760783.304876:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28311552, bytes: 4096 00000080:00200000:2.0:1789760783.304884:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.304892:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.304894:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28311552, 28315648) 0 0 00000080:00200000:2.0:1789760783.304898:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.304939:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6912, 6912] 00020000:00200000:2.0:1789760783.304947:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28311552 chunk: [28311552, 28315648) 28315648, 4096 00020000:00200000:2.0:1789760783.304951:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.304967:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6912, 6912] 00000008:00010000:2.0:1789760783.304981:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.304986:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760783.304994:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28311552, 28315648) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.305005:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28311552=0x1b00000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.305046:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.305049:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28315648 00000020:00010000:2.0:1789760783.305056:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.305060:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.305063:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.305065:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.305068:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.305075:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.305080:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.305093:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760783.305178:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 28311552<>28315648 00000100:00100000:3.0:1789760783.305378:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760783.305392:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512748416:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760783.341026:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974231 gen 3 00000008:00010000:3.0:1789760783.341065:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760783.341071:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.341079:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.341098:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760783.341132:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512748416:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760783.341220:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6912, 6912] 00000080:00200000:2.0:1789760783.341232:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.341237:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.341248:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28315648 i_blocks 55303 00000020:00200000:2.0:1789760783.341251:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.341259:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.341283:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28315648, count: 4096 00000080:00200000:2.0:1789760783.341289:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28315648, bytes: 4096 00000080:00200000:2.0:1789760783.341296:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.341302:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.341304:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28315648, 28319744) 0 0 00000080:00200000:2.0:1789760783.341308:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.341320:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6913, 6913] 00020000:00200000:2.0:1789760783.341327:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28315648 chunk: [28315648, 28319744) 28319744, 4096 00020000:00200000:2.0:1789760783.341330:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.341345:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6913, 6913] 00000008:00010000:2.0:1789760783.341359:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.341364:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760783.341373:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28315648, 28319744) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.341383:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28315648=0x1b01000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.341424:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.341427:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28319744 00000020:00010000:2.0:1789760783.341434:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.341439:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.341442:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.341445:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.341448:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.341454:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.341459:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.341472:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760783.341558:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 28315648<>28319744 00000100:00100000:3.0:1789760783.341651:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512748544:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760783.374959:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974232 gen 3 00000008:00010000:3.0:1789760783.374994:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760783.375000:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.375008:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.375028:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760783.375050:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512748544:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760783.375141:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6913, 6913] 00000080:00200000:2.0:1789760783.375155:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.375160:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.375171:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28319744 i_blocks 57351 00000020:00200000:2.0:1789760783.375174:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.375182:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.375206:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28319744, count: 4096 00000080:00200000:2.0:1789760783.375212:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28319744, bytes: 4096 00000080:00200000:2.0:1789760783.375220:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.375226:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.375228:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28319744, 28323840) 0 0 00000080:00200000:2.0:1789760783.375232:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.375244:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6914, 6914] 00020000:00200000:2.0:1789760783.375249:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28319744 chunk: [28319744, 28323840) 28323840, 4096 00020000:00200000:2.0:1789760783.375253:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.375265:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6914, 6914] 00000008:00010000:2.0:1789760783.375277:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.375281:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760783.375290:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28319744, 28323840) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.375300:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28319744=0x1b02000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.375342:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.375345:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28323840 00000020:00010000:2.0:1789760783.375351:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.375355:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.375358:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.375361:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.375364:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.375370:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.375376:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.375388:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760783.375475:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 28319744<>28323840 00000100:00100000:3.0:1789760783.375637:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512748672:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760783.408663:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974233 gen 3 00000008:00010000:3.0:1789760783.408698:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760783.408704:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.408712:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.408728:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760783.408750:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512748672:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760783.408888:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6914, 6914] 00000080:00200000:2.0:1789760783.408925:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.408930:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.408942:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28323840 i_blocks 57351 00000020:00200000:2.0:1789760783.408945:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.408953:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.408978:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28323840, count: 4096 00000080:00200000:2.0:1789760783.408984:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28323840, bytes: 4096 00000080:00200000:2.0:1789760783.408991:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.408998:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.409000:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28323840, 28327936) 0 0 00000080:00200000:2.0:1789760783.409004:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.409016:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6915, 6915] 00020000:00200000:2.0:1789760783.409023:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28323840 chunk: [28323840, 28327936) 28327936, 4096 00020000:00200000:2.0:1789760783.409026:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.409040:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6915, 6915] 00000008:00010000:2.0:1789760783.409053:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.409057:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760783.409065:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28323840, 28327936) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.409075:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28323840=0x1b03000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.409127:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.409131:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28327936 00000020:00010000:2.0:1789760783.409137:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.409140:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.409143:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.409146:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.409149:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.409155:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.409161:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.409175:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760783.409245:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 28323840<>28327936 00000100:00100000:3.0:1789760783.409305:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512748800:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760783.434052:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974234 gen 3 00000008:00010000:3.0:1789760783.434153:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760783.434160:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.434169:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.434255:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000080:00200000:2.0:1789760783.434279:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6915, 6915] 00000100:00100000:3.0:1789760783.434279:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512748800:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760783.434291:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.434296:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.434306:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28327936 i_blocks 57351 00000020:00200000:2.0:1789760783.434309:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.434316:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.434340:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28327936, count: 4096 00000080:00200000:2.0:1789760783.434346:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28327936, bytes: 4096 00000080:00200000:2.0:1789760783.434353:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.434360:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.434362:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28327936, 28332032) 0 0 00000080:00200000:2.0:1789760783.434366:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.434379:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6916, 6916] 00020000:00200000:2.0:1789760783.434385:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28327936 chunk: [28327936, 28332032) 28332032, 4096 00020000:00200000:2.0:1789760783.434389:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.434403:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6916, 6916] 00000008:00010000:2.0:1789760783.434415:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.434420:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760783.434429:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28327936, 28332032) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.434441:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28327936=0x1b04000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.434485:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.434488:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28332032 00000020:00010000:2.0:1789760783.434494:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.434499:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.434501:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.434505:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.434508:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.434516:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.434521:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.434535:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760783.434609:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 28327936<>28332032 00000100:00100000:3.0:1789760783.434704:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512748928:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760783.459972:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974235 gen 3 00000008:00010000:3.0:1789760783.460006:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760783.460013:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.460021:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.460038:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760783.460059:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512748928:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760783.460499:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6916, 6916] 00000080:00200000:2.0:1789760783.460520:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.460525:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.460536:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28332032 i_blocks 57351 00000020:00200000:2.0:1789760783.460540:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.460548:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.460573:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28332032, count: 4096 00000080:00200000:2.0:1789760783.460579:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28332032, bytes: 4096 00000080:00200000:2.0:1789760783.460747:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.460754:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.460756:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28332032, 28336128) 0 0 00000080:00200000:2.0:1789760783.460760:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.460772:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6917, 6917] 00020000:00200000:2.0:1789760783.460779:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28332032 chunk: [28332032, 28336128) 28336128, 4096 00020000:00200000:2.0:1789760783.460790:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.460804:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6917, 6917] 00000008:00010000:2.0:1789760783.460817:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.460822:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760783.460830:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28332032, 28336128) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.460839:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28332032=0x1b05000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.460880:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.460882:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28336128 00000020:00010000:2.0:1789760783.460888:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.460892:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.460895:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.460898:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.460922:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.460928:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.460934:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.460948:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760783.461022:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 28332032<>28336128 00000100:00100000:3.0:1789760783.461167:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760783.461181:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512749056:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760783.496966:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974236 gen 3 00000008:00010000:3.0:1789760783.497002:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760783.497009:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.497017:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.497035:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760783.497262:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512749056:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760783.497998:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6917, 6917] 00000080:00200000:2.0:1789760783.498013:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.498018:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.498031:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28336128 i_blocks 57351 00000020:00200000:2.0:1789760783.498034:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.498043:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.498068:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28336128, count: 4096 00000080:00200000:2.0:1789760783.498075:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28336128, bytes: 4096 00000080:00200000:2.0:1789760783.498083:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.498091:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.498093:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28336128, 28340224) 0 0 00000080:00200000:2.0:1789760783.498097:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.498119:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6918, 6918] 00020000:00200000:2.0:1789760783.498126:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28336128 chunk: [28336128, 28340224) 28340224, 4096 00020000:00200000:2.0:1789760783.498130:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.498146:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6918, 6918] 00000008:00010000:2.0:1789760783.498160:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.498164:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760783.498172:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28336128, 28340224) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.498181:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28336128=0x1b06000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.498229:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.498231:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28340224 00000020:00010000:2.0:1789760783.498237:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.498241:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.498244:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.498246:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.498248:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.498255:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.498260:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.498274:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760783.498349:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 28336128<>28340224 00000100:00100000:3.0:1789760783.499118:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760783.499140:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512749184:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760783.551948:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974237 gen 3 00000008:00010000:3.0:1789760783.551983:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760783.551989:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.551997:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.552016:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760783.552037:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512749184:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760783.552127:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6918, 6918] 00000080:00200000:2.0:1789760783.552139:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.552144:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.552155:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28340224 i_blocks 57351 00000020:00200000:2.0:1789760783.552158:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.552165:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.552190:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28340224, count: 4096 00000080:00200000:2.0:1789760783.552196:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28340224, bytes: 4096 00000080:00200000:2.0:1789760783.552203:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.552211:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.552213:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28340224, 28344320) 0 0 00000080:00200000:2.0:1789760783.552216:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.552230:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6919, 6919] 00020000:00200000:2.0:1789760783.552236:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28340224 chunk: [28340224, 28344320) 28344320, 4096 00020000:00200000:2.0:1789760783.552240:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.552256:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6919, 6919] 00000008:00010000:2.0:1789760783.552269:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.552275:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760783.552287:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28340224, 28344320) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.552299:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28340224=0x1b07000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.552341:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.552344:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28344320 00000020:00010000:2.0:1789760783.552350:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.552354:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.552357:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.552361:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.552364:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.552370:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.552377:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.552391:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760783.552468:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 28340224<>28344320 00000100:00100000:3.0:1789760783.552738:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760783.552752:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512749312:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760783.598628:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974238 gen 3 00000008:00010000:3.0:1789760783.598666:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760783.598672:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.598681:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.598701:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760783.598723:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512749312:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760783.599292:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6919, 6919] 00000080:00200000:2.0:1789760783.599307:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.599312:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.599324:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28344320 i_blocks 57351 00000020:00200000:2.0:1789760783.599327:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.599336:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.599361:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28344320, count: 4096 00000080:00200000:2.0:1789760783.599367:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28344320, bytes: 4096 00000080:00200000:2.0:1789760783.599375:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.599381:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.599383:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28344320, 28348416) 0 0 00000080:00200000:2.0:1789760783.599387:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.599400:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6920, 6920] 00020000:00200000:2.0:1789760783.599407:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28344320 chunk: [28344320, 28348416) 28348416, 4096 00020000:00200000:2.0:1789760783.599411:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.599426:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6920, 6920] 00000008:00010000:2.0:1789760783.599439:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.599444:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760783.599452:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28344320, 28348416) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.599462:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28344320=0x1b08000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.599505:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.599508:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28348416 00000020:00010000:2.0:1789760783.599514:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.599518:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.599520:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.599524:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.599526:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.599532:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.599537:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.599552:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760783.599630:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 28344320<>28348416 00000100:00100000:3.0:1789760783.602040:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760783.602078:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512749440:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760783.634326:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974239 gen 3 00000008:00010000:3.0:1789760783.634361:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760783.634380:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.634389:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.634411:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760783.634436:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512749440:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760783.634521:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6920, 6920] 00000080:00200000:2.0:1789760783.634532:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.634538:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.634549:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28348416 i_blocks 57351 00000020:00200000:2.0:1789760783.634553:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.634561:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.634585:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28348416, count: 4096 00000080:00200000:2.0:1789760783.634591:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28348416, bytes: 4096 00000080:00200000:2.0:1789760783.634600:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.634607:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.634609:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28348416, 28352512) 0 0 00000080:00200000:2.0:1789760783.634613:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.634626:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6921, 6921] 00020000:00200000:2.0:1789760783.634635:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28348416 chunk: [28348416, 28352512) 28352512, 4096 00020000:00200000:2.0:1789760783.634638:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.634652:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6921, 6921] 00000008:00010000:2.0:1789760783.634664:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.634669:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760783.634678:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28348416, 28352512) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.634688:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28348416=0x1b09000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.634729:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.634731:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28352512 00000020:00010000:2.0:1789760783.634738:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.634742:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.634745:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.634748:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.634751:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.634758:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.634763:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.634775:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760783.634844:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 28348416<>28352512 00000100:00100000:3.0:1789760783.635828:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760783.635847:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512749568:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760783.645061:0:38:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1789760783.645079:0:38:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760783.645104:0:38:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1789760783.670765:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974240 gen 3 00000008:00010000:2.0:1789760783.674231:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:2.0:1789760783.674241:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760783.674250:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760783.675093:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:2.0:1789760783.675129:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512749568:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760783.677742:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6921, 6921] 00000080:00200000:0.0:1789760783.677764:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760783.677772:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760783.677786:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28352512 i_blocks 57351 00000020:00200000:0.0:1789760783.677791:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760783.677806:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760783.677834:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28352512, count: 4096 00000080:00200000:0.0:1789760783.677840:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28352512, bytes: 4096 00000080:00200000:0.0:1789760783.677851:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760783.677858:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760783.677860:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28352512, 28356608) 0 0 00000080:00200000:0.0:1789760783.677864:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760783.677880:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6922, 6922] 00020000:00200000:0.0:1789760783.677888:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28352512 chunk: [28352512, 28356608) 28356608, 4096 00020000:00200000:0.0:1789760783.677891:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760783.677927:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6922, 6922] 00000008:00010000:0.0:1789760783.677943:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760783.677949:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760783.677959:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28352512, 28356608) append 0 append lockpos 0 00000080:00200000:0.0:1789760783.677971:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28352512=0x1b0a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760783.678750:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760783.678755:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28356608 00000020:00010000:0.0:1789760783.678764:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760783.678768:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.678771:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.678774:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760783.678777:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760783.678784:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760783.678790:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760783.678815:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cca80 00000008:00100000:0.0:1789760783.678892:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148a80 - object 0x240000400:5 offset 28352512<>28356608 00000100:00100000:1.0:1789760783.680214:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760783.685795:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512749696:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760783.734063:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974241 gen 3 00000008:00010000:1.0:1789760783.734100:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760783.734119:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760783.734129:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760783.734152:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760783.734176:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512749696:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760783.734317:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6922, 6922] 00000080:00200000:2.0:1789760783.734330:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760783.734335:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760783.734349:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28356608 i_blocks 57351 00000020:00200000:2.0:1789760783.734353:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760783.734361:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760783.734389:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28356608, count: 4096 00000080:00200000:2.0:1789760783.734395:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28356608, bytes: 4096 00000080:00200000:2.0:1789760783.734403:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760783.734410:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760783.734412:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28356608, 28360704) 0 0 00000080:00200000:2.0:1789760783.734416:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760783.734428:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6923, 6923] 00020000:00200000:2.0:1789760783.734435:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28356608 chunk: [28356608, 28360704) 28360704, 4096 00020000:00200000:2.0:1789760783.734439:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760783.734458:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6923, 6923] 00000008:00010000:2.0:1789760783.734473:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760783.734478:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760783.734490:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28356608, 28360704) append 0 append lockpos 0 00000080:00200000:2.0:1789760783.734502:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28356608=0x1b0b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760783.734547:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760783.734550:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28360704 00000020:00010000:2.0:1789760783.734557:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760783.734560:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.734563:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760783.734566:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760783.734569:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760783.734576:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760783.734582:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760783.734596:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760783.734689:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 28356608<>28360704 00000100:00100000:3.0:1789760783.734763:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512749824:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760783.778628:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974242 gen 3 00000008:00010000:3.0:1789760783.778794:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760783.778800:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760783.778811:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760783.778895:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760783.778961:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512749824:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760783.780823:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6923, 6923] 00000080:00200000:0.0:1789760783.780844:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760783.780850:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760783.780863:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28360704 i_blocks 57351 00000020:00200000:0.0:1789760783.780867:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760783.780880:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760783.780923:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28360704, count: 4096 00000080:00200000:0.0:1789760783.780930:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28360704, bytes: 4096 00000080:00200000:0.0:1789760783.780942:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760783.780949:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760783.780951:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28360704, 28364800) 0 0 00000080:00200000:0.0:1789760783.780955:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760783.780969:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6924, 6924] 00020000:00200000:0.0:1789760783.780976:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28360704 chunk: [28360704, 28364800) 28364800, 4096 00020000:00200000:0.0:1789760783.780992:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760783.781011:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6924, 6924] 00000008:00010000:0.0:1789760783.781028:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760783.781032:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760783.781042:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28360704, 28364800) append 0 append lockpos 0 00000080:00200000:0.0:1789760783.781056:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28360704=0x1b0c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760783.781125:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760783.781128:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28364800 00000020:00010000:0.0:1789760783.781135:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760783.781139:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.781142:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.781155:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760783.781158:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760783.781165:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760783.781171:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760783.781193:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccf00 00000008:00100000:0.0:1789760783.781294:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148700 - object 0x240000400:5 offset 28360704<>28364800 00000100:00100000:1.0:1789760783.781376:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512749952:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760783.832419:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974243 gen 3 00000008:00010000:1.0:1789760783.832458:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760783.832464:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760783.832473:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760783.832497:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760783.832521:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512749952:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760783.832694:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6924, 6924] 00000080:00200000:0.0:1789760783.832718:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760783.832724:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760783.832741:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28364800 i_blocks 57351 00000020:00200000:0.0:1789760783.832745:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760783.832754:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760783.832780:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28364800, count: 4096 00000080:00200000:0.0:1789760783.832786:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28364800, bytes: 4096 00000080:00200000:0.0:1789760783.832794:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760783.832801:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760783.832803:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28364800, 28368896) 0 0 00000080:00200000:0.0:1789760783.832807:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760783.832820:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6925, 6925] 00020000:00200000:0.0:1789760783.832826:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28364800 chunk: [28364800, 28368896) 28368896, 4096 00020000:00200000:0.0:1789760783.832830:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760783.832844:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6925, 6925] 00000008:00010000:0.0:1789760783.833091:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760783.833098:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760783.833116:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28364800, 28368896) append 0 append lockpos 0 00000080:00200000:0.0:1789760783.833127:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28364800=0x1b0d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760783.833171:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760783.833174:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28368896 00000020:00010000:0.0:1789760783.833181:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760783.833184:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.833188:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.833191:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760783.833193:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760783.833199:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760783.833205:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760783.833219:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc000 00000008:00100000:0.0:1789760783.833435:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514ad80 - object 0x240000400:5 offset 28364800<>28368896 00000100:00100000:1.0:1789760783.833503:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512750080:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760783.880424:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974244 gen 3 00000008:00010000:1.0:1789760783.880464:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760783.880472:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760783.880481:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760783.880499:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760783.880522:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512750080:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760783.880722:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6925, 6925] 00000080:00200000:0.0:1789760783.880735:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760783.880740:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760783.880753:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28368896 i_blocks 57351 00000020:00200000:0.0:1789760783.880756:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760783.880764:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760783.880791:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28368896, count: 4096 00000080:00200000:0.0:1789760783.880797:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28368896, bytes: 4096 00000080:00200000:0.0:1789760783.880805:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760783.880813:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760783.880815:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28368896, 28372992) 0 0 00000080:00200000:0.0:1789760783.880819:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760783.880833:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6926, 6926] 00020000:00200000:0.0:1789760783.880839:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28368896 chunk: [28368896, 28372992) 28372992, 4096 00020000:00200000:0.0:1789760783.880842:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760783.880855:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6926, 6926] 00000008:00010000:0.0:1789760783.880869:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760783.880873:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760783.880882:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28368896, 28372992) append 0 append lockpos 0 00000080:00200000:0.0:1789760783.880892:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28368896=0x1b0e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760783.880969:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760783.880972:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28372992 00000020:00010000:0.0:1789760783.880979:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760783.880983:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.880986:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.880989:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760783.880992:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760783.880998:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760783.881005:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760783.881019:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000008:00100000:0.0:1789760783.881122:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148e00 - object 0x240000400:5 offset 28368896<>28372992 00000100:00100000:1.0:1789760783.883605:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512750208:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760783.920963:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974245 gen 3 00000008:00010000:1.0:1789760783.921001:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760783.921006:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760783.921016:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760783.921040:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760783.921064:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512750208:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760783.921292:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6926, 6926] 00000080:00200000:0.0:1789760783.921305:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760783.921309:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760783.921323:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28372992 i_blocks 57351 00000020:00200000:0.0:1789760783.921326:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760783.921335:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760783.921363:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28372992, count: 4096 00000080:00200000:0.0:1789760783.921370:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28372992, bytes: 4096 00000080:00200000:0.0:1789760783.921382:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760783.921389:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760783.921390:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28372992, 28377088) 0 0 00000080:00200000:0.0:1789760783.921394:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760783.921406:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6927, 6927] 00020000:00200000:0.0:1789760783.921412:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28372992 chunk: [28372992, 28377088) 28377088, 4096 00020000:00200000:0.0:1789760783.921416:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760783.921430:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6927, 6927] 00000008:00010000:0.0:1789760783.921443:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760783.921448:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760783.921456:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28372992, 28377088) append 0 append lockpos 0 00000080:00200000:0.0:1789760783.921465:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28372992=0x1b0f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760783.921506:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760783.921508:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28377088 00000020:00010000:0.0:1789760783.921514:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760783.921518:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.921521:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760783.921525:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760783.921527:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760783.921532:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760783.921537:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760783.921552:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc780 00000008:00100000:0.0:1789760783.921626:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a680 - object 0x240000400:5 offset 28372992<>28377088 00000100:00100000:0.0:1789760783.921665:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760783.921677:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512750336:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760783.969946:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974246 gen 3 00000008:00010000:0.0:1789760783.970052:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:0.0:1789760783.970059:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760783.970068:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760783.970156:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:0.0:1789760783.970182:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512750336:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760783.971363:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6927, 6927] 00000080:00200000:1.0:1789760783.971378:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760783.971384:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760783.971398:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28377088 i_blocks 57351 00000020:00200000:1.0:1789760783.971402:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760783.971413:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760783.971436:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28377088, count: 4096 00000080:00200000:1.0:1789760783.971443:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28377088, bytes: 4096 00000080:00200000:1.0:1789760783.971454:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760783.971769:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760783.971771:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28377088, 28381184) 0 0 00000080:00200000:1.0:1789760783.971775:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760783.971789:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6928, 6928] 00020000:00200000:1.0:1789760783.971795:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28377088 chunk: [28377088, 28381184) 28381184, 4096 00020000:00200000:1.0:1789760783.971799:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760783.971816:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6928, 6928] 00000008:00010000:1.0:1789760783.971891:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760783.971896:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760783.971995:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28377088, 28381184) append 0 append lockpos 0 00000080:00200000:1.0:1789760783.972007:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28377088=0x1b10000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760783.972056:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760783.972059:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28381184 00000020:00010000:1.0:1789760783.972065:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760783.972069:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760783.972072:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760783.972075:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760783.972078:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760783.972084:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760783.972088:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760783.972119:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760783.972195:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97480 - object 0x240000400:5 offset 28377088<>28381184 00000100:00100000:0.0:1789760783.972257:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512750464:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760784.020008:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974247 gen 3 00000008:00010000:0.0:1789760784.020064:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760784.020073:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760784.020084:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760784.020096:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760784.020135:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512750464:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760784.020344:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6928, 6928] 00000080:00200000:1.0:1789760784.020362:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760784.020369:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760784.020386:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28381184 i_blocks 57351 00000020:00200000:1.0:1789760784.020390:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760784.020400:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760784.020436:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28381184, count: 4096 00000080:00200000:1.0:1789760784.020453:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28381184, bytes: 4096 00000080:00200000:1.0:1789760784.020465:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760784.020474:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760784.020476:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28381184, 28385280) 0 0 00000080:00200000:1.0:1789760784.020480:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760784.020496:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6929, 6929] 00020000:00200000:1.0:1789760784.020505:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28381184 chunk: [28381184, 28385280) 28385280, 4096 00020000:00200000:1.0:1789760784.020510:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760784.020960:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6929, 6929] 00000008:00010000:1.0:1789760784.020979:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760784.020985:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760784.020997:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28381184, 28385280) append 0 append lockpos 0 00000080:00200000:1.0:1789760784.021121:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28381184=0x1b11000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760784.021801:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760784.021805:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28385280 00000020:00010000:1.0:1789760784.021814:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760784.021817:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.021820:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.021823:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760784.021825:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760784.021834:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760784.021840:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760784.021866:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760784.021990:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 28381184<>28385280 00000100:00100000:0.0:1789760784.022293:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760784.022315:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512750592:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760784.117753:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974248 gen 3 00000008:00010000:0.0:1789760784.117807:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760784.117816:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760784.117827:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760784.117851:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760784.117885:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512750592:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760784.118505:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6929, 6929] 00000080:00200000:1.0:1789760784.118828:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760784.118836:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760784.118856:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28385280 i_blocks 57351 00000020:00200000:1.0:1789760784.118862:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760784.118875:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760784.118982:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28385280, count: 4096 00000080:00200000:1.0:1789760784.118993:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28385280, bytes: 4096 00000080:00200000:1.0:1789760784.119009:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760784.119019:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760784.119021:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28385280, 28389376) 0 0 00000080:00200000:1.0:1789760784.119026:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760784.119045:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6930, 6930] 00020000:00200000:1.0:1789760784.119054:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28385280 chunk: [28385280, 28389376) 28389376, 4096 00020000:00200000:1.0:1789760784.119059:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760784.119081:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6930, 6930] 00000008:00010000:1.0:1789760784.119099:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760784.119116:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760784.119129:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28385280, 28389376) append 0 append lockpos 0 00000080:00200000:1.0:1789760784.119144:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28385280=0x1b12000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760784.119219:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760784.119223:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28389376 00000020:00010000:1.0:1789760784.119231:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760784.119235:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.119238:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.119242:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760784.119245:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760784.119253:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760784.119259:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760784.119281:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760784.119413:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 28385280<>28389376 00000100:00100000:0.0:1789760784.120136:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512750720:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760784.160236:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760784.243367:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974249 gen 3 00000008:00010000:0.0:1789760784.243572:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760784.243582:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760784.243596:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760784.243619:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760784.243650:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512750720:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760784.243662:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6930, 6930] 00000080:00200000:1.0:1789760784.243680:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760784.243686:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760784.243702:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28389376 i_blocks 57351 00000020:00200000:1.0:1789760784.243706:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760784.243716:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760784.243754:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28389376, count: 4096 00000080:00200000:1.0:1789760784.243763:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28389376, bytes: 4096 00000080:00200000:1.0:1789760784.243780:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760784.243790:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760784.243792:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28389376, 28393472) 0 0 00000080:00200000:1.0:1789760784.243797:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760784.243816:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6931, 6931] 00020000:00200000:1.0:1789760784.243826:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28389376 chunk: [28389376, 28393472) 28393472, 4096 00020000:00200000:1.0:1789760784.243831:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760784.243851:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6931, 6931] 00000008:00010000:1.0:1789760784.243870:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760784.243876:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760784.243889:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28389376, 28393472) append 0 append lockpos 0 00000080:00200000:1.0:1789760784.243923:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28389376=0x1b13000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760784.243980:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760784.243984:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28393472 00000020:00010000:1.0:1789760784.243993:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760784.243996:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.243999:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.244003:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760784.244006:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760784.244014:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760784.244021:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760784.244037:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760784.244169:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 28389376<>28393472 00000100:00100000:0.0:1789760784.244271:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512750848:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760784.285422:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760784.285438:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760784.285450:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760784.285482:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760784.285526:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512750848:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760784.285920:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6931, 6931] 00000080:00200000:1.0:1789760784.285938:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760784.285944:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760784.285959:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28393472 i_blocks 57351 00000020:00200000:1.0:1789760784.285963:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760784.285973:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760784.286009:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28393472, count: 4096 00000080:00200000:1.0:1789760784.286017:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28393472, bytes: 4096 00000080:00200000:1.0:1789760784.286032:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760784.286043:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760784.286045:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28393472, 28397568) 0 0 00000080:00200000:1.0:1789760784.286050:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760784.286067:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6932, 6932] 00020000:00200000:1.0:1789760784.286077:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28393472 chunk: [28393472, 28397568) 28397568, 4096 00020000:00200000:1.0:1789760784.286082:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760784.286113:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6932, 6932] 00000008:00010000:1.0:1789760784.286131:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760784.286137:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760784.286147:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28393472, 28397568) append 0 append lockpos 0 00000080:00200000:1.0:1789760784.286161:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28393472=0x1b14000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760784.286214:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760784.286217:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28397568 00000020:00010000:1.0:1789760784.286225:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760784.286229:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.286232:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.286236:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760784.286239:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760784.286245:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760784.286252:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760784.286268:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760784.286387:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94380 - object 0x240000400:5 offset 28393472<>28397568 00000100:00100000:0.0:1789760784.286457:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512750976:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760784.324018:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974250 gen 3 00000100:00100000:0.0:1789760784.324063:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005c94380 x1876699512750976/t4294974251(4294974251) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760800 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760784.324123:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760784.324131:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760784.324141:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760784.324163:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760784.324187:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512750976:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760784.324285:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6932, 6932] 00000080:00200000:1.0:1789760784.324302:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760784.324307:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760784.324323:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28397568 i_blocks 57351 00000020:00200000:1.0:1789760784.324327:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760784.324338:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760784.324375:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28397568, count: 4096 00000080:00200000:1.0:1789760784.324383:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28397568, bytes: 4096 00000080:00200000:1.0:1789760784.324392:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760784.324400:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760784.324402:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28397568, 28401664) 0 0 00000080:00200000:1.0:1789760784.324470:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760784.324489:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6933, 6933] 00020000:00200000:1.0:1789760784.324556:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28397568 chunk: [28397568, 28401664) 28401664, 4096 00020000:00200000:1.0:1789760784.324561:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760784.324584:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6933, 6933] 00000008:00010000:1.0:1789760784.324604:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760784.324609:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760784.324620:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28397568, 28401664) append 0 append lockpos 0 00000080:00200000:1.0:1789760784.324636:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28397568=0x1b15000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760784.324886:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760784.324889:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28401664 00000020:00010000:1.0:1789760784.325570:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760784.325582:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.325587:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.325591:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760784.325595:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760784.325608:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760784.325617:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760784.325648:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760784.326050:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 28397568<>28401664 00000100:00100000:0.0:1789760784.326145:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760784.326304:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512751104:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760784.369139:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974251 gen 3 00000100:00100000:0.0:1789760784.369190:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005c95500 x1876699512751104/t4294974252(4294974252) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760800 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760784.369246:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760784.369254:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760784.369265:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760784.369294:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760784.369331:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512751104:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760784.369638:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6933, 6933] 00000080:00200000:1.0:1789760784.369720:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760784.369728:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760784.369743:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28401664 i_blocks 57351 00000020:00200000:1.0:1789760784.369747:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760784.369757:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760784.369793:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28401664, count: 4096 00000080:00200000:1.0:1789760784.369801:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28401664, bytes: 4096 00000080:00200000:1.0:1789760784.369811:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760784.369820:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760784.369823:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28401664, 28405760) 0 0 00000080:00200000:1.0:1789760784.369828:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760784.369855:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6934, 6934] 00020000:00200000:1.0:1789760784.369864:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28401664 chunk: [28401664, 28405760) 28405760, 4096 00020000:00200000:1.0:1789760784.369869:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760784.369890:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6934, 6934] 00000008:00010000:1.0:1789760784.369938:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760784.369945:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760784.369957:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28401664, 28405760) append 0 append lockpos 0 00000080:00200000:1.0:1789760784.369971:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28401664=0x1b16000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760784.370035:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760784.370038:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28405760 00000020:00010000:1.0:1789760784.370046:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760784.370050:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.370053:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.370057:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760784.370060:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760784.370069:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760784.370076:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760784.370734:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760784.370854:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94700 - object 0x240000400:5 offset 28401664<>28405760 00000100:00100000:1.0:1789760784.370945:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760784.370969:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512751232:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760784.427605:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974253 gen 3 00000008:00010000:0.0:1789760784.428432:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760784.428444:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760784.428456:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760784.428484:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760784.428519:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512751232:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760784.428702:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6934, 6934] 00000080:00200000:2.0:1789760784.428719:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.428726:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.428742:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28405760 i_blocks 57351 00000020:00200000:2.0:1789760784.428746:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.428756:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.428794:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28405760, count: 4096 00000080:00200000:2.0:1789760784.428802:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28405760, bytes: 4096 00000080:00200000:2.0:1789760784.428816:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.428826:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.428828:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28405760, 28409856) 0 0 00000080:00200000:2.0:1789760784.428833:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.428852:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6935, 6935] 00020000:00200000:2.0:1789760784.428861:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28405760 chunk: [28405760, 28409856) 28409856, 4096 00020000:00200000:2.0:1789760784.428866:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.428886:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6935, 6935] 00000008:00010000:2.0:1789760784.428925:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.428932:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760784.428943:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28405760, 28409856) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.428958:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28405760=0x1b17000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.429012:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.429014:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28409856 00000020:00010000:2.0:1789760784.429023:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.429027:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.429030:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.429033:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.429035:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.429043:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.429050:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.429069:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760784.429189:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 28405760<>28409856 00000100:00100000:2.0:1789760784.429245:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512751360:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760784.471371:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974254 gen 3 00000008:00010000:3.0:1789760784.471408:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760784.471414:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.471423:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.471442:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760784.471465:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512751360:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760784.471585:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6935, 6935] 00000080:00200000:2.0:1789760784.471598:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.471603:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.471615:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28409856 i_blocks 57351 00000020:00200000:2.0:1789760784.471618:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.471626:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.471653:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28409856, count: 4096 00000080:00200000:2.0:1789760784.471659:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28409856, bytes: 4096 00000080:00200000:2.0:1789760784.471666:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.471673:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.471676:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28409856, 28413952) 0 0 00000080:00200000:2.0:1789760784.471679:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.471691:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6936, 6936] 00020000:00200000:2.0:1789760784.471698:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28409856 chunk: [28409856, 28413952) 28413952, 4096 00020000:00200000:2.0:1789760784.471703:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.471717:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6936, 6936] 00000008:00010000:2.0:1789760784.471731:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.471735:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760784.471744:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28409856, 28413952) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.471755:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28409856=0x1b18000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.471817:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.471820:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28413952 00000020:00010000:2.0:1789760784.471827:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.471830:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.471833:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.471836:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.471839:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.471845:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.471850:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.471864:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760784.471956:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 28409856<>28413952 00000100:00100000:3.0:1789760784.472135:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760784.472148:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512751488:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760784.512567:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760784.512578:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.512585:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.512602:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760784.512628:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512751488:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760784.512800:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6936, 6936] 00000080:00200000:2.0:1789760784.512815:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.512820:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.512832:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28413952 i_blocks 57351 00000020:00200000:2.0:1789760784.512836:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.512845:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.512869:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28413952, count: 4096 00000080:00200000:2.0:1789760784.512875:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28413952, bytes: 4096 00000080:00200000:2.0:1789760784.512883:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.512890:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.512892:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28413952, 28418048) 0 0 00000080:00200000:2.0:1789760784.512896:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.512932:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6937, 6937] 00020000:00200000:2.0:1789760784.512939:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28413952 chunk: [28413952, 28418048) 28418048, 4096 00020000:00200000:2.0:1789760784.512943:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.512959:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6937, 6937] 00000008:00010000:2.0:1789760784.512973:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.512977:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760784.512985:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28413952, 28418048) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.512995:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28413952=0x1b19000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.513036:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.513039:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28418048 00000020:00010000:2.0:1789760784.513046:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.513049:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.513052:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.513055:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.513057:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.513063:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.513068:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.513082:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760784.513172:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 28413952<>28418048 00000100:00100000:3.0:1789760784.513386:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512751616:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760784.557203:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974255 gen 3 00000100:00100000:3.0:1789760784.557233:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368d180 x1876699512751616/t4294974256(4294974256) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760800 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760784.557271:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760784.557277:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.557284:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.557300:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760784.557324:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512751616:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760784.557405:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6937, 6937] 00000080:00200000:2.0:1789760784.557416:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.557421:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.557431:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28418048 i_blocks 57351 00000020:00200000:2.0:1789760784.557435:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.557443:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.557468:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28418048, count: 4096 00000080:00200000:2.0:1789760784.557475:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28418048, bytes: 4096 00000080:00200000:2.0:1789760784.557482:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.557488:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.557490:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28418048, 28422144) 0 0 00000080:00200000:2.0:1789760784.557493:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.557596:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6938, 6938] 00020000:00200000:2.0:1789760784.557603:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28418048 chunk: [28418048, 28422144) 28422144, 4096 00020000:00200000:2.0:1789760784.557608:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.557624:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6938, 6938] 00000008:00010000:2.0:1789760784.557637:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.557641:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760784.557660:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28418048, 28422144) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.557672:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28418048=0x1b1a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.557715:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.557718:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28422144 00000020:00010000:2.0:1789760784.557725:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.557728:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.557731:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.557734:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.557737:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.557743:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.557749:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.557764:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760784.557838:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 28418048<>28422144 00000100:00100000:3.0:1789760784.557955:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760784.557969:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512751744:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760784.608429:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974257 gen 3 00000008:00010000:3.0:1789760784.608553:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760784.608562:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.608573:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.608595:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760784.608638:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512751744:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760784.608794:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6938, 6938] 00000080:00200000:2.0:1789760784.608811:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.608818:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.608834:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28422144 i_blocks 57351 00000020:00200000:2.0:1789760784.608839:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.608849:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.608884:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28422144, count: 4096 00000080:00200000:2.0:1789760784.608891:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28422144, bytes: 4096 00000080:00200000:2.0:1789760784.608931:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.608941:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.608943:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28422144, 28426240) 0 0 00000080:00200000:2.0:1789760784.608947:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.608965:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6939, 6939] 00020000:00200000:2.0:1789760784.608973:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28422144 chunk: [28422144, 28426240) 28426240, 4096 00020000:00200000:2.0:1789760784.608978:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.608997:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6939, 6939] 00000008:00010000:2.0:1789760784.609016:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.609022:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760784.609034:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28422144, 28426240) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.609048:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28422144=0x1b1b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.609113:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.609116:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28426240 00000020:00010000:2.0:1789760784.609123:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.609126:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.609129:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.609132:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.609135:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.609142:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.609148:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.609164:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760784.609268:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 28422144<>28426240 00000100:00100000:2.0:1789760784.609318:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760784.609336:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512751872:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760784.657159:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974258 gen 3 00000008:00010000:3.0:1789760784.657215:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760784.657222:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.657233:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.657255:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760784.657286:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512751872:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760784.657471:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6939, 6939] 00000080:00200000:2.0:1789760784.657499:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.657506:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.657523:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28426240 i_blocks 57351 00000020:00200000:2.0:1789760784.657527:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.657537:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.657575:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28426240, count: 4096 00000080:00200000:2.0:1789760784.657583:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28426240, bytes: 4096 00000080:00200000:2.0:1789760784.657597:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.657606:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.657608:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28426240, 28430336) 0 0 00000080:00200000:2.0:1789760784.657613:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.657644:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6940, 6940] 00020000:00200000:2.0:1789760784.657653:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28426240 chunk: [28426240, 28430336) 28430336, 4096 00020000:00200000:2.0:1789760784.657658:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.657680:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6940, 6940] 00000008:00010000:2.0:1789760784.657700:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.657706:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760784.657718:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28426240, 28430336) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.657736:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28426240=0x1b1c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.657795:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.657798:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28430336 00000020:00010000:2.0:1789760784.657807:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.657811:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.657815:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.657819:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.657822:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.657831:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.657838:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.657857:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760784.658003:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 28426240<>28430336 00000100:00100000:3.0:1789760784.658068:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512752000:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760784.711130:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974259 gen 3 00000008:00010000:3.0:1789760784.711253:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760784.711263:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.711275:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.711388:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760784.711425:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512752000:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760784.711541:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6940, 6940] 00000080:00200000:2.0:1789760784.711559:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760784.711566:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760784.711582:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28430336 i_blocks 57351 00000020:00200000:2.0:1789760784.711586:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760784.711597:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760784.711635:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28430336, count: 4096 00000080:00200000:2.0:1789760784.711642:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28430336, bytes: 4096 00000080:00200000:2.0:1789760784.711658:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760784.711667:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760784.711669:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28430336, 28434432) 0 0 00000080:00200000:2.0:1789760784.711675:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760784.711693:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6941, 6941] 00020000:00200000:2.0:1789760784.711702:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28430336 chunk: [28430336, 28434432) 28434432, 4096 00020000:00200000:2.0:1789760784.711707:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760784.711728:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6941, 6941] 00000008:00010000:2.0:1789760784.711747:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760784.711753:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760784.711765:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28430336, 28434432) append 0 append lockpos 0 00000080:00200000:2.0:1789760784.711780:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28430336=0x1b1d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760784.711838:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760784.711841:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28434432 00000020:00010000:2.0:1789760784.712014:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760784.712019:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.712022:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760784.712026:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760784.712029:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760784.712038:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760784.712045:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760784.712065:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760784.712180:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 28430336<>28434432 00000100:00100000:3.0:1789760784.712306:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760784.712374:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512752128:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760784.762349:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974260 gen 3 00000008:00010000:3.0:1789760784.762401:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760784.762410:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760784.762422:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760784.762450:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760784.762485:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512752128:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760784.763052:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6941, 6941] 00000080:00200000:1.0:1789760784.763070:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760784.763076:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760784.763093:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28434432 i_blocks 57351 00000020:00200000:1.0:1789760784.763098:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760784.763118:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760784.763154:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28434432, count: 4096 00000080:00200000:1.0:1789760784.763161:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28434432, bytes: 4096 00000080:00200000:1.0:1789760784.763173:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760784.763181:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760784.763184:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28434432, 28438528) 0 0 00000080:00200000:1.0:1789760784.763188:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760784.763205:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6942, 6942] 00020000:00200000:1.0:1789760784.763214:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28434432 chunk: [28434432, 28438528) 28438528, 4096 00020000:00200000:1.0:1789760784.763218:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760784.763239:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6942, 6942] 00000008:00010000:1.0:1789760784.763258:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760784.763265:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760784.763278:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28434432, 28438528) append 0 append lockpos 0 00000080:00200000:1.0:1789760784.763293:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28434432=0x1b1e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760784.763359:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760784.763362:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28438528 00000020:00010000:1.0:1789760784.763371:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760784.763375:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.763377:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.763380:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760784.763383:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760784.763391:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760784.763398:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760784.763419:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760784.763536:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 28434432<>28438528 00000100:00100000:0.0:1789760784.763682:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512752256:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760784.807451:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760784.807469:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760784.807481:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760784.807506:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760784.807540:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512752256:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760784.807659:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6942, 6942] 00000080:00200000:1.0:1789760784.807675:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760784.807681:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760784.807701:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28438528 i_blocks 57351 00000020:00200000:1.0:1789760784.807706:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760784.807717:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760784.807754:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28438528, count: 4096 00000080:00200000:1.0:1789760784.807763:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28438528, bytes: 4096 00000080:00200000:1.0:1789760784.807775:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760784.807784:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760784.807787:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28438528, 28442624) 0 0 00000080:00200000:1.0:1789760784.807791:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760784.807810:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6943, 6943] 00020000:00200000:1.0:1789760784.807819:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28438528 chunk: [28438528, 28442624) 28442624, 4096 00020000:00200000:1.0:1789760784.807825:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760784.807845:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6943, 6943] 00000008:00010000:1.0:1789760784.807863:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760784.807869:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760784.807881:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28438528, 28442624) append 0 append lockpos 0 00000080:00200000:1.0:1789760784.807897:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28438528=0x1b1f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760784.807987:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760784.807990:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28442624 00000020:00010000:1.0:1789760784.807999:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760784.808003:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.808007:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.808010:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760784.808013:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760784.808021:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760784.808028:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760784.808050:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760784.808171:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 28438528<>28442624 00000100:00100000:0.0:1789760784.809087:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760784.809126:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512752384:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760784.850637:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974262 gen 3 00000008:00010000:0.0:1789760784.850708:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760784.850717:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760784.850728:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760784.850754:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760784.850787:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512752384:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760784.852951:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6943, 6943] 00000080:00200000:1.0:1789760784.852977:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760784.852985:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760784.853002:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28442624 i_blocks 57351 00000020:00200000:1.0:1789760784.853010:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760784.853023:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760784.853058:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28442624, count: 4096 00000080:00200000:1.0:1789760784.853067:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28442624, bytes: 4096 00000080:00200000:1.0:1789760784.853080:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760784.853090:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760784.853093:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28442624, 28446720) 0 0 00000080:00200000:1.0:1789760784.853097:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760784.853136:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6944, 6944] 00020000:00200000:1.0:1789760784.853146:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28442624 chunk: [28442624, 28446720) 28446720, 4096 00020000:00200000:1.0:1789760784.853150:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760784.853173:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6944, 6944] 00000008:00010000:1.0:1789760784.853192:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760784.853198:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760784.853221:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28442624, 28446720) append 0 append lockpos 0 00000080:00200000:1.0:1789760784.853237:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28442624=0x1b20000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760784.853307:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760784.853310:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28446720 00000020:00010000:1.0:1789760784.853318:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760784.853322:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.853325:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.853328:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760784.853331:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760784.853339:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760784.853346:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760784.853369:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760784.853482:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95c00 - object 0x240000400:5 offset 28442624<>28446720 00000100:00100000:0.0:1789760784.854319:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760784.854353:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512752512:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760784.919742:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760784.919753:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760784.919762:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760784.919784:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760784.919807:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512752512:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760784.919924:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6944, 6944] 00000080:00200000:1.0:1789760784.919937:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760784.919942:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760784.919953:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28446720 i_blocks 57351 00000020:00200000:1.0:1789760784.919957:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760784.919968:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760784.919993:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28446720, count: 4096 00000080:00200000:1.0:1789760784.919999:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28446720, bytes: 4096 00000080:00200000:1.0:1789760784.920006:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760784.920013:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760784.920015:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28446720, 28450816) 0 0 00000080:00200000:1.0:1789760784.920018:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760784.920031:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6945, 6945] 00020000:00200000:1.0:1789760784.920037:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28446720 chunk: [28446720, 28450816) 28450816, 4096 00020000:00200000:1.0:1789760784.920041:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760784.920054:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6945, 6945] 00000008:00010000:1.0:1789760784.920067:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760784.920071:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760784.920080:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28446720, 28450816) append 0 append lockpos 0 00000080:00200000:1.0:1789760784.920091:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28446720=0x1b21000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760784.920144:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760784.920147:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28450816 00000020:00010000:1.0:1789760784.920152:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760784.920155:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.920157:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.920160:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760784.920162:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760784.920168:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760784.920173:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760784.920187:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760784.920265:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 28446720<>28450816 00000100:00100000:0.0:1789760784.920406:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512752640:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760784.959692:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974264 gen 3 00000008:00010000:0.0:1789760784.959737:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760784.959744:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760784.959752:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760784.959774:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760784.959797:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512752640:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760784.959889:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6945, 6945] 00000080:00200000:1.0:1789760784.959932:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760784.959938:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760784.959949:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28450816 i_blocks 57351 00000020:00200000:1.0:1789760784.959953:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760784.959961:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760784.959986:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28450816, count: 4096 00000080:00200000:1.0:1789760784.959992:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28450816, bytes: 4096 00000080:00200000:1.0:1789760784.959999:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760784.960006:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760784.960008:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28450816, 28454912) 0 0 00000080:00200000:1.0:1789760784.960012:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760784.960025:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6946, 6946] 00020000:00200000:1.0:1789760784.960032:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28450816 chunk: [28450816, 28454912) 28454912, 4096 00020000:00200000:1.0:1789760784.960035:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760784.960052:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6946, 6946] 00000008:00010000:1.0:1789760784.960064:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760784.960069:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760784.960077:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28450816, 28454912) append 0 append lockpos 0 00000080:00200000:1.0:1789760784.960088:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28450816=0x1b22000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760784.960139:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760784.960142:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28454912 00000020:00010000:1.0:1789760784.960149:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760784.960152:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.960155:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760784.960158:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760784.960161:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760784.960166:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760784.960172:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760784.960186:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760784.960261:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 28450816<>28454912 00000100:00100000:0.0:1789760784.960446:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512752768:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760785.012444:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760785.012456:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760785.012465:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760785.012489:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760785.012513:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512752768:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760785.012933:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6946, 6946] 00000080:00200000:1.0:1789760785.012945:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760785.012950:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760785.012960:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28454912 i_blocks 57351 00000020:00200000:1.0:1789760785.012964:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760785.012972:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760785.012996:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28454912, count: 4096 00000080:00200000:1.0:1789760785.013003:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28454912, bytes: 4096 00000080:00200000:1.0:1789760785.013012:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760785.013018:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760785.013020:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28454912, 28459008) 0 0 00000080:00200000:1.0:1789760785.013023:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760785.013037:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6947, 6947] 00020000:00200000:1.0:1789760785.013043:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28454912 chunk: [28454912, 28459008) 28459008, 4096 00020000:00200000:1.0:1789760785.013047:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760785.013061:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6947, 6947] 00000008:00010000:1.0:1789760785.013074:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760785.013078:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760785.013088:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28454912, 28459008) append 0 append lockpos 0 00000080:00200000:1.0:1789760785.013211:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28454912=0x1b23000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760785.013256:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760785.013258:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28459008 00000020:00010000:1.0:1789760785.013264:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760785.013414:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.013417:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.013420:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760785.013423:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760785.013430:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760785.013436:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760785.013451:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760785.013526:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 28454912<>28459008 00000100:00100000:0.0:1789760785.013712:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512752896:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760785.056819:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974266 gen 3 00000008:00010000:0.0:1789760785.056862:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760785.056869:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760785.056877:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760785.056986:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760785.057004:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512752896:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760785.057136:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6947, 6947] 00000080:00200000:1.0:1789760785.057149:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760785.057153:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760785.057164:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28459008 i_blocks 57351 00000020:00200000:1.0:1789760785.057168:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760785.057176:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760785.057202:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28459008, count: 4096 00000080:00200000:1.0:1789760785.057209:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28459008, bytes: 4096 00000080:00200000:1.0:1789760785.057216:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760785.057223:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760785.057225:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28459008, 28463104) 0 0 00000080:00200000:1.0:1789760785.057229:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760785.057241:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6948, 6948] 00020000:00200000:1.0:1789760785.057248:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28459008 chunk: [28459008, 28463104) 28463104, 4096 00020000:00200000:1.0:1789760785.057252:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760785.057266:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6948, 6948] 00000008:00010000:1.0:1789760785.057280:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760785.057284:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760785.057293:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28459008, 28463104) append 0 append lockpos 0 00000080:00200000:1.0:1789760785.057306:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28459008=0x1b24000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760785.057351:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760785.057354:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28463104 00000020:00010000:1.0:1789760785.057360:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760785.057363:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.057366:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.057452:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760785.057456:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760785.057463:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760785.057469:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760785.057483:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760785.057562:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97100 - object 0x240000400:5 offset 28459008<>28463104 00000100:00100000:0.0:1789760785.057622:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512753024:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760785.089814:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974267 gen 3 00000008:00010000:0.0:1789760785.089847:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760785.089853:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760785.089861:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760785.089881:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000080:00200000:1.0:1789760785.089985:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6948, 6948] 00000080:00200000:1.0:1789760785.089999:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760785.090004:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760785.090016:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28463104 i_blocks 57351 00000020:00200000:1.0:1789760785.090020:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760785.090027:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760785.090057:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28463104, count: 4096 00000080:00200000:1.0:1789760785.090064:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28463104, bytes: 4096 00000080:00200000:1.0:1789760785.090072:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760785.090079:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760785.090081:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28463104, 28467200) 0 0 00000080:00200000:1.0:1789760785.090085:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760785.090106:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6949, 6949] 00020000:00200000:1.0:1789760785.090113:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28463104 chunk: [28463104, 28467200) 28467200, 4096 00020000:00200000:1.0:1789760785.090117:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760785.090132:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6949, 6949] 00000008:00010000:1.0:1789760785.090144:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760785.090149:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760785.090157:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28463104, 28467200) append 0 append lockpos 0 00000080:00200000:1.0:1789760785.090167:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28463104=0x1b25000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760785.090212:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760785.090214:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28467200 00000020:00010000:1.0:1789760785.090220:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760785.090223:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.090226:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.090229:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760785.090233:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760785.090239:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760785.090244:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760785.090259:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760785.090328:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 28463104<>28467200 00000100:00100000:1.0:1789760785.090367:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512753152:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760785.090550:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512753024:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760785.117828:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974268 gen 3 00000008:00010000:1.0:1789760785.117861:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760785.117868:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760785.117877:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760785.118038:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760785.118062:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512753152:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760785.118191:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6949, 6949] 00000080:00200000:2.0:1789760785.118202:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760785.118207:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760785.118218:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28467200 i_blocks 57351 00000020:00200000:2.0:1789760785.118220:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760785.118228:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760785.118253:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28467200, count: 4096 00000080:00200000:2.0:1789760785.118258:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28467200, bytes: 4096 00000080:00200000:2.0:1789760785.118335:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760785.118342:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760785.118345:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28467200, 28471296) 0 0 00000080:00200000:2.0:1789760785.118349:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760785.118363:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6950, 6950] 00020000:00200000:2.0:1789760785.118369:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28467200 chunk: [28467200, 28471296) 28471296, 4096 00020000:00200000:2.0:1789760785.118373:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760785.118388:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6950, 6950] 00000008:00010000:2.0:1789760785.118402:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760785.118407:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760785.118415:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28467200, 28471296) append 0 append lockpos 0 00000080:00200000:2.0:1789760785.118426:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28467200=0x1b26000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760785.118474:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760785.118477:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28471296 00000020:00010000:2.0:1789760785.118483:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760785.118486:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.118551:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.118555:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760785.118558:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760785.118565:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760785.118572:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760785.118586:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760785.118721:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 28467200<>28471296 00000100:00100000:2.0:1789760785.118768:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760785.118782:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512753280:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760785.155335:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974269 gen 3 00000008:00010000:3.0:1789760785.155372:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760785.155379:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.155388:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.156189:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760785.156218:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512753280:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760785.157397:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6950, 6950] 00000080:00200000:2.0:1789760785.157414:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760785.157420:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760785.157434:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28471296 i_blocks 57351 00000020:00200000:2.0:1789760785.157439:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760785.157450:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760785.157476:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28471296, count: 4096 00000080:00200000:2.0:1789760785.157483:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28471296, bytes: 4096 00000080:00200000:2.0:1789760785.157506:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760785.157515:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760785.157517:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28471296, 28475392) 0 0 00000080:00200000:2.0:1789760785.157522:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760785.157537:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6951, 6951] 00020000:00200000:2.0:1789760785.157545:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28471296 chunk: [28471296, 28475392) 28475392, 4096 00020000:00200000:2.0:1789760785.157548:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760785.157565:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6951, 6951] 00000008:00010000:2.0:1789760785.157593:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760785.157598:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760785.157607:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28471296, 28475392) append 0 append lockpos 0 00000080:00200000:2.0:1789760785.157619:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28471296=0x1b27000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760785.157685:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760785.157688:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28475392 00000020:00010000:2.0:1789760785.157694:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760785.157697:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.157700:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.157703:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760785.157706:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760785.157713:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760785.157719:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760785.157741:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760785.157826:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 28471296<>28475392 00000100:00100000:3.0:1789760785.158046:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512753408:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760785.181387:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512753664:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760785.181481:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1789760785.181489:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e6cf7267-9b19-462a-a263-14115df15dab:2360:1876699512753536:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760785.181520:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2360:1876699512753792:192.168.203.156@tcp:400:kworker.0 00010000:00010000:1.0:1789760785.181807:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760785.188953:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e6cf7267-9b19-462a-a263-14115df15dab:2360:1876699512753536:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760785.188985:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2360:1876699512753792:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760785.188998:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512753664:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760785.189008:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974270 gen 3 00000008:00010000:3.0:1789760785.189035:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760785.189041:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.189049:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.189069:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760785.189090:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512753408:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760785.189256:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6951, 6951] 00000080:00200000:2.0:1789760785.189278:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760785.189283:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760785.189295:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28475392 i_blocks 57351 00000020:00200000:2.0:1789760785.189298:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760785.189306:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760785.189332:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28475392, count: 4096 00000080:00200000:2.0:1789760785.189338:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28475392, bytes: 4096 00000080:00200000:2.0:1789760785.189345:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760785.189353:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760785.189355:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28475392, 28479488) 0 0 00000080:00200000:2.0:1789760785.189359:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760785.189372:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6952, 6952] 00020000:00200000:2.0:1789760785.189378:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28475392 chunk: [28475392, 28479488) 28479488, 4096 00020000:00200000:2.0:1789760785.189383:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760785.189399:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6952, 6952] 00000008:00010000:2.0:1789760785.189414:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760785.189418:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760785.189427:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28475392, 28479488) append 0 append lockpos 0 00000080:00200000:2.0:1789760785.189439:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28475392=0x1b28000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760785.189484:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760785.189487:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28479488 00000020:00010000:2.0:1789760785.189494:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760785.189497:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.189500:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.189503:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760785.189505:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760785.189513:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760785.189519:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760785.189532:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760785.189607:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 28475392<>28479488 00000100:00100000:3.0:1789760785.189735:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760785.189747:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512753920:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760785.249493:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760785.249504:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.249514:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.249539:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760785.249564:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512753920:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760785.249872:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6952, 6952] 00000080:00200000:2.0:1789760785.249885:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760785.249890:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760785.249928:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28479488 i_blocks 57351 00000020:00200000:2.0:1789760785.249932:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760785.249941:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760785.249962:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28479488, count: 4096 00000080:00200000:2.0:1789760785.249968:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28479488, bytes: 4096 00000080:00200000:2.0:1789760785.249979:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760785.249986:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760785.249988:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28479488, 28483584) 0 0 00000080:00200000:2.0:1789760785.249992:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760785.250003:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6953, 6953] 00020000:00200000:2.0:1789760785.250010:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28479488 chunk: [28479488, 28483584) 28483584, 4096 00020000:00200000:2.0:1789760785.250014:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760785.250027:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6953, 6953] 00000008:00010000:2.0:1789760785.250041:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760785.250045:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760785.250054:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28479488, 28483584) append 0 append lockpos 0 00000080:00200000:2.0:1789760785.250064:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28479488=0x1b29000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760785.250246:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760785.250249:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28483584 00000020:00010000:2.0:1789760785.250256:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760785.250259:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.250262:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.250265:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760785.250267:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760785.250274:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760785.250279:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760785.250292:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760785.250371:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 28479488<>28483584 00000100:00100000:3.0:1789760785.250461:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760785.250477:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512754048:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760785.306572:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974272 gen 3 00000008:00010000:3.0:1789760785.306620:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760785.306626:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.306634:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.306654:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760785.306679:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512754048:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760785.306760:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6953, 6953] 00000080:00200000:2.0:1789760785.306772:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760785.306777:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760785.306793:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28483584 i_blocks 57351 00000020:00200000:2.0:1789760785.306796:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760785.306805:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760785.306832:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28483584, count: 4096 00000080:00200000:2.0:1789760785.306838:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28483584, bytes: 4096 00000080:00200000:2.0:1789760785.306846:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760785.306858:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760785.306860:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28483584, 28487680) 0 0 00000080:00200000:2.0:1789760785.306864:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760785.306879:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6954, 6954] 00020000:00200000:2.0:1789760785.306886:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28483584 chunk: [28483584, 28487680) 28487680, 4096 00020000:00200000:2.0:1789760785.306889:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760785.306931:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6954, 6954] 00000008:00010000:2.0:1789760785.306946:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760785.306952:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760785.306962:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28483584, 28487680) append 0 append lockpos 0 00000080:00200000:2.0:1789760785.306971:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28483584=0x1b2a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760785.307016:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760785.307018:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28487680 00000020:00010000:2.0:1789760785.307025:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760785.307028:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.307031:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.307034:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760785.307037:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760785.307043:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760785.307049:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760785.307062:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760785.307146:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 28483584<>28487680 00000100:00100000:3.0:1789760785.307345:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760785.307359:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512754176:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760785.362782:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974273 gen 3 00000008:00010000:3.0:1789760785.362822:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760785.362829:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.362837:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.362945:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760785.362965:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512754176:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760785.363028:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6954, 6954] 00000080:00200000:2.0:1789760785.363040:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760785.363045:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760785.363056:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28487680 i_blocks 57351 00000020:00200000:2.0:1789760785.363059:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760785.363067:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760785.363091:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28487680, count: 4096 00000080:00200000:2.0:1789760785.363107:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28487680, bytes: 4096 00000080:00200000:2.0:1789760785.363115:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760785.363122:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760785.363124:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28487680, 28491776) 0 0 00000080:00200000:2.0:1789760785.363128:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760785.363142:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6955, 6955] 00020000:00200000:2.0:1789760785.363149:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28487680 chunk: [28487680, 28491776) 28491776, 4096 00020000:00200000:2.0:1789760785.363153:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760785.363168:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6955, 6955] 00000008:00010000:2.0:1789760785.363182:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760785.363186:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760785.363195:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28487680, 28491776) append 0 append lockpos 0 00000080:00200000:2.0:1789760785.363205:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28487680=0x1b2b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760785.363247:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760785.363249:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28491776 00000020:00010000:2.0:1789760785.363256:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760785.363259:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.363262:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.363265:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760785.363268:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760785.363274:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760785.363280:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760785.363292:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760785.363364:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 28487680<>28491776 00000100:00100000:3.0:1789760785.363418:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512754304:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760785.408051:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974274 gen 3 00000008:00010000:3.0:1789760785.408086:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760785.408092:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.408110:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.408129:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760785.408160:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512754304:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760785.408368:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6955, 6955] 00000080:00200000:2.0:1789760785.408380:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760785.408385:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760785.408397:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28491776 i_blocks 57351 00000020:00200000:2.0:1789760785.408400:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760785.408408:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760785.408433:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28491776, count: 4096 00000080:00200000:2.0:1789760785.408439:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28491776, bytes: 4096 00000080:00200000:2.0:1789760785.408447:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760785.408454:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760785.408456:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28491776, 28495872) 0 0 00000080:00200000:2.0:1789760785.408460:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760785.408473:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6956, 6956] 00020000:00200000:2.0:1789760785.408479:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28491776 chunk: [28491776, 28495872) 28495872, 4096 00020000:00200000:2.0:1789760785.408483:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760785.408511:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6956, 6956] 00000008:00010000:2.0:1789760785.408525:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760785.408530:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760785.408538:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28491776, 28495872) append 0 append lockpos 0 00000080:00200000:2.0:1789760785.408549:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28491776=0x1b2c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760785.408594:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760785.408597:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28495872 00000020:00010000:2.0:1789760785.408604:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760785.408607:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.408610:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.408613:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760785.408616:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760785.408622:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760785.408629:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760785.408642:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760785.408714:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 28491776<>28495872 00000100:00100000:3.0:1789760785.408782:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512754432:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760785.452070:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974275 gen 3 00000008:00010000:3.0:1789760785.452118:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760785.452124:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.452197:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.452215:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760785.452242:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512754432:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760785.452329:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6956, 6956] 00000080:00200000:2.0:1789760785.452342:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760785.452347:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760785.452359:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28495872 i_blocks 57351 00000020:00200000:2.0:1789760785.452363:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760785.452371:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760785.452398:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28495872, count: 4096 00000080:00200000:2.0:1789760785.452404:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28495872, bytes: 4096 00000080:00200000:2.0:1789760785.452413:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760785.452420:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760785.452422:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28495872, 28499968) 0 0 00000080:00200000:2.0:1789760785.452426:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760785.452440:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6957, 6957] 00020000:00200000:2.0:1789760785.452447:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28495872 chunk: [28495872, 28499968) 28499968, 4096 00020000:00200000:2.0:1789760785.452450:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760785.452466:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6957, 6957] 00000008:00010000:2.0:1789760785.452480:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760785.452485:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760785.452494:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28495872, 28499968) append 0 append lockpos 0 00000080:00200000:2.0:1789760785.452504:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28495872=0x1b2d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760785.452548:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760785.452551:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28499968 00000020:00010000:2.0:1789760785.452556:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760785.452559:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.452562:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.452565:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760785.452567:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760785.452573:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760785.452579:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760785.452592:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760785.452755:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 28495872<>28499968 00000100:00100000:3.0:1789760785.452855:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512754560:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760785.501410:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760785.501423:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.501432:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.501452:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760785.501475:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512754560:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760785.501550:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6957, 6957] 00000080:00200000:2.0:1789760785.501564:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760785.501569:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760785.501581:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28499968 i_blocks 57351 00000020:00200000:2.0:1789760785.501585:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760785.501593:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760785.501618:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28499968, count: 4096 00000080:00200000:2.0:1789760785.501624:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28499968, bytes: 4096 00000080:00200000:2.0:1789760785.501632:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760785.501639:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760785.501641:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28499968, 28504064) 0 0 00000080:00200000:2.0:1789760785.501646:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760785.501669:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6958, 6958] 00020000:00200000:2.0:1789760785.501677:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28499968 chunk: [28499968, 28504064) 28504064, 4096 00020000:00200000:2.0:1789760785.501681:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760785.501696:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6958, 6958] 00000008:00010000:2.0:1789760785.501710:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760785.501715:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760785.501735:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28499968, 28504064) append 0 append lockpos 0 00000080:00200000:2.0:1789760785.501746:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28499968=0x1b2e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760785.501791:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760785.501794:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28504064 00000020:00010000:2.0:1789760785.501801:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760785.501804:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.501807:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.501810:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760785.501813:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760785.501820:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760785.501825:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760785.501840:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760785.501970:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 28499968<>28504064 00000100:00100000:2.0:1789760785.502008:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760785.502020:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512754688:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760785.554152:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974277 gen 3 00000008:00010000:3.0:1789760785.554199:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760785.554206:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.554214:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.554240:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760785.554264:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512754688:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760785.554360:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6958, 6958] 00000080:00200000:1.0:1789760785.554383:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760785.554388:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760785.554401:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28504064 i_blocks 57351 00000020:00200000:1.0:1789760785.554405:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760785.554413:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760785.554441:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28504064, count: 4096 00000080:00200000:1.0:1789760785.554448:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28504064, bytes: 4096 00000080:00200000:1.0:1789760785.554455:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760785.554463:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760785.554465:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28504064, 28508160) 0 0 00000080:00200000:1.0:1789760785.554469:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760785.554482:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6959, 6959] 00020000:00200000:1.0:1789760785.554499:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28504064 chunk: [28504064, 28508160) 28508160, 4096 00020000:00200000:1.0:1789760785.554502:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760785.554517:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6959, 6959] 00000008:00010000:1.0:1789760785.554530:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760785.554535:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760785.554543:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28504064, 28508160) append 0 append lockpos 0 00000080:00200000:1.0:1789760785.554554:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28504064=0x1b2f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760785.554597:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760785.554600:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28508160 00000020:00010000:1.0:1789760785.554606:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760785.554609:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.554612:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760785.554615:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760785.554618:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760785.554624:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760785.554629:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760785.554642:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760785.554716:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 28504064<>28508160 00000100:00100000:1.0:1789760785.554756:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760785.555079:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512754816:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760785.588303:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974278 gen 3 00000008:00010000:1.0:1789760785.588337:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760785.588343:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760785.588350:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760785.588479:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760785.588503:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512754816:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760785.588588:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6959, 6959] 00000080:00200000:0.0:1789760785.588600:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760785.588605:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760785.588617:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28508160 i_blocks 57351 00000020:00200000:0.0:1789760785.588621:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760785.588629:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760785.588655:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28508160, count: 4096 00000080:00200000:0.0:1789760785.588661:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28508160, bytes: 4096 00000080:00200000:0.0:1789760785.588668:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760785.588675:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760785.588677:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28508160, 28512256) 0 0 00000080:00200000:0.0:1789760785.588680:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760785.588692:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6960, 6960] 00020000:00200000:0.0:1789760785.588699:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28508160 chunk: [28508160, 28512256) 28512256, 4096 00020000:00200000:0.0:1789760785.588702:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760785.588716:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6960, 6960] 00000008:00010000:0.0:1789760785.588731:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760785.588735:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760785.588754:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28508160, 28512256) append 0 append lockpos 0 00000080:00200000:0.0:1789760785.588765:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28508160=0x1b30000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760785.588820:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760785.588823:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28512256 00000020:00010000:0.0:1789760785.588829:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760785.588832:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.588836:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.588839:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760785.588842:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760785.588848:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760785.588853:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760785.588866:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc780 00000008:00100000:0.0:1789760785.588976:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a680 - object 0x240000400:5 offset 28508160<>28512256 00000100:00100000:0.0:1789760785.589005:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760785.589017:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512754944:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760785.637409:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:0.0:1789760785.637421:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760785.637430:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760785.637571:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:0.0:1789760785.637596:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512754944:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760785.637897:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6960, 6960] 00000080:00200000:3.0:1789760785.638044:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760785.638050:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760785.638062:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28512256 i_blocks 57351 00000020:00200000:3.0:1789760785.638065:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760785.638073:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760785.638106:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28512256, count: 4096 00000080:00200000:3.0:1789760785.638112:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28512256, bytes: 4096 00000080:00200000:3.0:1789760785.638122:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760785.638129:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760785.638131:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28512256, 28516352) 0 0 00000080:00200000:3.0:1789760785.638135:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760785.638149:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6961, 6961] 00020000:00200000:3.0:1789760785.638155:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28512256 chunk: [28512256, 28516352) 28516352, 4096 00020000:00200000:3.0:1789760785.638159:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760785.638172:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6961, 6961] 00000008:00010000:3.0:1789760785.638313:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760785.638318:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760785.638327:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28512256, 28516352) append 0 append lockpos 0 00000080:00200000:3.0:1789760785.638338:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28512256=0x1b31000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760785.638387:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760785.638390:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28516352 00000020:00010000:3.0:1789760785.638398:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760785.638401:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.638404:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.638407:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760785.638410:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760785.638415:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760785.638421:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760785.638437:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760785.638692:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad880 - object 0x240000400:5 offset 28512256<>28516352 00000100:00100000:2.0:1789760785.638950:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512755072:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760785.680957:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974280 gen 3 00000008:00010000:3.0:1789760785.681002:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:3.0:1789760785.681008:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.681017:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.681086:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:3.0:1789760785.681117:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512755072:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760785.681455:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6961, 6961] 00000080:00200000:0.0:1789760785.681467:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760785.681473:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760785.681546:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28516352 i_blocks 57351 00000020:00200000:0.0:1789760785.681551:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760785.681682:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760785.681709:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28516352, count: 4096 00000080:00200000:0.0:1789760785.681715:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28516352, bytes: 4096 00000080:00200000:0.0:1789760785.681724:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760785.681731:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760785.681733:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28516352, 28520448) 0 0 00000080:00200000:0.0:1789760785.681736:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760785.681749:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6962, 6962] 00020000:00200000:0.0:1789760785.681755:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28516352 chunk: [28516352, 28520448) 28520448, 4096 00020000:00200000:0.0:1789760785.681758:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760785.681872:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6962, 6962] 00000008:00010000:0.0:1789760785.681887:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760785.681892:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760785.681934:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28516352, 28520448) append 0 append lockpos 0 00000080:00200000:0.0:1789760785.681945:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28516352=0x1b32000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760785.681992:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760785.681994:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28520448 00000020:00010000:0.0:1789760785.682000:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760785.682003:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.682005:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.682008:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760785.682010:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760785.682017:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760785.682023:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760785.682041:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc780 00000008:00100000:0.0:1789760785.682131:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a300 - object 0x240000400:5 offset 28516352<>28520448 00000100:00100000:1.0:1789760785.682267:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512755200:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760785.731193:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974281 gen 3 00000008:00010000:1.0:1789760785.731228:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760785.731234:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760785.731244:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760785.731265:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760785.731287:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512755200:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760785.731496:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6962, 6962] 00000080:00200000:0.0:1789760785.731509:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760785.731515:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760785.731528:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28520448 i_blocks 57351 00000020:00200000:0.0:1789760785.731531:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760785.731539:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760785.731567:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28520448, count: 4096 00000080:00200000:0.0:1789760785.731574:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28520448, bytes: 4096 00000080:00200000:0.0:1789760785.731582:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760785.731589:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760785.731591:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28520448, 28524544) 0 0 00000080:00200000:0.0:1789760785.731595:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760785.731608:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6963, 6963] 00020000:00200000:0.0:1789760785.731615:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28520448 chunk: [28520448, 28524544) 28524544, 4096 00020000:00200000:0.0:1789760785.731618:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760785.731633:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6963, 6963] 00000008:00010000:0.0:1789760785.731646:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760785.731650:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760785.731660:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28520448, 28524544) append 0 append lockpos 0 00000080:00200000:0.0:1789760785.731670:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28520448=0x1b33000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760785.731717:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760785.731720:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28524544 00000020:00010000:0.0:1789760785.731726:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760785.731729:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.731732:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.731735:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760785.731738:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760785.731744:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760785.731749:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760785.731773:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccd80 00000008:00100000:0.0:1789760785.731858:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149880 - object 0x240000400:5 offset 28520448<>28524544 00000100:00100000:0.0:1789760785.731897:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512755328:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760785.826275:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974282 gen 3 00000008:00010000:0.0:1789760785.826311:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:0.0:1789760785.826317:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760785.826326:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760785.826350:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:0.0:1789760785.826447:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512755328:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760785.826542:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6963, 6963] 00000080:00200000:3.0:1789760785.826555:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760785.826561:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760785.826573:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28524544 i_blocks 57351 00000020:00200000:3.0:1789760785.826576:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760785.826584:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760785.826704:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28524544, count: 4096 00000080:00200000:3.0:1789760785.826711:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28524544, bytes: 4096 00000080:00200000:3.0:1789760785.826720:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760785.826727:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760785.826729:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28524544, 28528640) 0 0 00000080:00200000:3.0:1789760785.826733:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760785.826746:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6964, 6964] 00020000:00200000:3.0:1789760785.826753:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28524544 chunk: [28524544, 28528640) 28528640, 4096 00020000:00200000:3.0:1789760785.826757:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760785.826772:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6964, 6964] 00000008:00010000:3.0:1789760785.826785:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760785.826789:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760785.826798:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28524544, 28528640) append 0 append lockpos 0 00000080:00200000:3.0:1789760785.826807:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28524544=0x1b34000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760785.827274:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760785.827278:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28528640 00000020:00010000:3.0:1789760785.827286:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760785.827290:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.827294:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760785.827298:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760785.827301:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760785.827309:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760785.827315:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760785.827334:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760785.828274:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad880 - object 0x240000400:5 offset 28524544<>28528640 00000100:00100000:3.0:1789760785.828333:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760785.828352:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512755456:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760785.916611:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974283 gen 3 00000008:00010000:3.0:1789760785.916660:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:3.0:1789760785.916669:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.916682:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.916701:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:3.0:1789760785.916729:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512755456:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760785.916945:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6964, 6964] 00000080:00200000:2.0:1789760785.916962:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760785.916967:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760785.916985:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28528640 i_blocks 57351 00000020:00200000:2.0:1789760785.916998:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760785.917010:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760785.917045:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28528640, count: 4096 00000080:00200000:2.0:1789760785.917053:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28528640, bytes: 4096 00000080:00200000:2.0:1789760785.917063:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760785.917073:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760785.917075:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28528640, 28532736) 0 0 00000080:00200000:2.0:1789760785.917080:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760785.917107:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6965, 6965] 00020000:00200000:2.0:1789760785.917118:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28528640 chunk: [28528640, 28532736) 28532736, 4096 00020000:00200000:2.0:1789760785.917122:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760785.917142:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6965, 6965] 00000008:00010000:2.0:1789760785.917160:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760785.917167:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760785.917179:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28528640, 28532736) append 0 append lockpos 0 00000080:00200000:2.0:1789760785.917193:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28528640=0x1b35000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760785.917253:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760785.917256:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28532736 00000020:00010000:2.0:1789760785.917265:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760785.917269:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.917271:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760785.917275:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760785.917278:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760785.917285:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760785.917293:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760785.917309:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760785.917419:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 28528640<>28532736 00000100:00100000:2.0:1789760785.917462:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760785.917480:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512755584:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760785.993890:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974284 gen 3 00000008:00010000:3.0:1789760785.993957:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760785.993966:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760785.993977:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760785.994069:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760785.994119:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512755584:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760785.996260:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6965, 6965] 00000080:00200000:0.0:1789760785.996280:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760785.996285:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760785.996299:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28532736 i_blocks 57351 00000020:00200000:0.0:1789760785.996303:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760785.996316:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760785.996341:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28532736, count: 4096 00000080:00200000:0.0:1789760785.996349:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28532736, bytes: 4096 00000080:00200000:0.0:1789760785.996360:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760785.996369:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760785.996371:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28532736, 28536832) 0 0 00000080:00200000:0.0:1789760785.996375:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760785.996390:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6966, 6966] 00020000:00200000:0.0:1789760785.996397:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28532736 chunk: [28532736, 28536832) 28536832, 4096 00020000:00200000:0.0:1789760785.996401:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760785.996417:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6966, 6966] 00000008:00010000:0.0:1789760785.996433:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760785.996439:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760785.996448:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28532736, 28536832) append 0 append lockpos 0 00000080:00200000:0.0:1789760785.996460:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28532736=0x1b36000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760785.996512:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760785.996515:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28536832 00000020:00010000:0.0:1789760785.996522:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760785.996525:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.996528:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760785.996531:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760785.996534:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760785.996541:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760785.996547:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760785.996567:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccd80 00000008:00100000:0.0:1789760785.996653:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149880 - object 0x240000400:5 offset 28532736<>28536832 00000100:00100000:0.0:1789760785.996701:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760785.996720:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512755712:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760786.063355:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974285 gen 3 00000008:00010000:0.0:1789760786.063392:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:0.0:1789760786.063399:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760786.063408:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760786.063431:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:0.0:1789760786.063456:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512755712:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760786.063696:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6966, 6966] 00000080:00200000:3.0:1789760786.063717:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760786.063723:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760786.063739:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28536832 i_blocks 57351 00000020:00200000:3.0:1789760786.063743:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760786.063756:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760786.063790:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28536832, count: 4096 00000080:00200000:3.0:1789760786.063798:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28536832, bytes: 4096 00000080:00200000:3.0:1789760786.063811:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760786.063831:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760786.063833:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28536832, 28540928) 0 0 00000080:00200000:3.0:1789760786.063837:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760786.063852:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6967, 6967] 00020000:00200000:3.0:1789760786.063860:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28536832 chunk: [28536832, 28540928) 28540928, 4096 00020000:00200000:3.0:1789760786.063865:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760786.063886:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6967, 6967] 00000008:00010000:3.0:1789760786.063943:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760786.063951:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760786.063963:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28536832, 28540928) append 0 append lockpos 0 00000080:00200000:3.0:1789760786.063980:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28536832=0x1b37000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760786.064179:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760786.064182:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28540928 00000020:00010000:3.0:1789760786.064192:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760786.064196:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.064198:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.064202:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760786.064204:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760786.064212:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760786.064219:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760786.064239:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760786.064352:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad880 - object 0x240000400:5 offset 28536832<>28540928 00000100:00100000:3.0:1789760786.064402:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760786.064424:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512755840:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760786.106814:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:3.0:1789760786.106840:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.106852:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.106877:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:3.0:1789760786.106931:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512755840:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760786.107128:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6967, 6967] 00000080:00200000:0.0:1789760786.107328:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.107334:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.107347:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28540928 i_blocks 57351 00000020:00200000:0.0:1789760786.107352:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.107363:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.107388:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28540928, count: 4096 00000080:00200000:0.0:1789760786.107395:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28540928, bytes: 4096 00000080:00200000:0.0:1789760786.107406:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.107413:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.107414:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28540928, 28545024) 0 0 00000080:00200000:0.0:1789760786.107418:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.107430:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6968, 6968] 00020000:00200000:0.0:1789760786.107437:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28540928 chunk: [28540928, 28545024) 28545024, 4096 00020000:00200000:0.0:1789760786.107440:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.107456:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6968, 6968] 00000008:00010000:0.0:1789760786.107471:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.107476:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760786.107484:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28540928, 28545024) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.107494:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28540928=0x1b38000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.107543:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.107546:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28545024 00000020:00010000:0.0:1789760786.107553:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.107556:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.107559:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.107562:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.107564:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.107570:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.107576:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.107595:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccd80 00000008:00100000:0.0:1789760786.107677:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149880 - object 0x240000400:5 offset 28540928<>28545024 00000100:00100000:1.0:1789760786.107755:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512755968:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760786.163639:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974286 gen 3 00000100:00100000:1.0:1789760786.163670:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005149880 x1876699512755968/t4294974287(4294974287) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760802 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760786.163706:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760786.163711:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.163720:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.163745:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760786.163766:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512755968:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760786.163951:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6968, 6968] 00000080:00200000:0.0:1789760786.163966:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.163972:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.163986:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28545024 i_blocks 57351 00000020:00200000:0.0:1789760786.163990:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.164000:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.164027:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28545024, count: 4096 00000080:00200000:0.0:1789760786.164034:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28545024, bytes: 4096 00000080:00200000:0.0:1789760786.164044:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.164051:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.164053:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28545024, 28549120) 0 0 00000080:00200000:0.0:1789760786.164057:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.164072:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6969, 6969] 00020000:00200000:0.0:1789760786.164079:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28545024 chunk: [28545024, 28549120) 28549120, 4096 00020000:00200000:0.0:1789760786.164083:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.164111:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6969, 6969] 00000008:00010000:0.0:1789760786.164125:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.164130:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760786.164139:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28545024, 28549120) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.164271:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28545024=0x1b39000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.164317:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.164320:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28549120 00000020:00010000:0.0:1789760786.164327:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.164330:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.164333:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.164337:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.164339:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.164345:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.164350:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.164366:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc180 00000008:00100000:0.0:1789760786.164439:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148380 - object 0x240000400:5 offset 28545024<>28549120 00000100:00100000:1.0:1789760786.164503:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512756096:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760786.207693:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760786.208017:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974287 gen 3 00000100:00100000:1.0:1789760786.208044:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005148380 x1876699512756096/t4294974288(4294974288) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760802 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760786.208080:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760786.208085:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.208105:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.208125:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760786.208151:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512756096:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760786.208267:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6969, 6969] 00000080:00200000:3.0:1789760786.208279:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760786.208284:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760786.208294:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28549120 i_blocks 57351 00000020:00200000:3.0:1789760786.208298:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760786.208305:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760786.208329:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28549120, count: 4096 00000080:00200000:3.0:1789760786.208336:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28549120, bytes: 4096 00000080:00200000:3.0:1789760786.208347:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760786.208354:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760786.208356:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28549120, 28553216) 0 0 00000080:00200000:3.0:1789760786.208360:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760786.208374:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6970, 6970] 00020000:00200000:3.0:1789760786.208381:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28549120 chunk: [28549120, 28553216) 28553216, 4096 00020000:00200000:3.0:1789760786.208385:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760786.208401:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6970, 6970] 00000008:00010000:3.0:1789760786.208415:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760786.208420:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760786.208429:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28549120, 28553216) append 0 append lockpos 0 00000080:00200000:3.0:1789760786.208439:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28549120=0x1b3a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760786.208477:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760786.208480:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28553216 00000020:00010000:3.0:1789760786.208485:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760786.208488:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.208491:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.208494:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760786.208497:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760786.208502:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760786.208507:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760786.208554:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:1.0:1789760786.208632:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95180 - object 0x240000400:5 offset 28549120<>28553216 00000100:00100000:0.0:1789760786.208697:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512756224:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760786.256105:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974289 gen 3 00000008:00010000:0.0:1789760786.256150:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:0.0:1789760786.256156:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760786.256165:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760786.256243:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000100:00100000:0.0:1789760786.256271:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512756224:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760786.256364:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6970, 6970] 00000080:00200000:3.0:1789760786.256376:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760786.256381:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760786.256392:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28553216 i_blocks 57351 00000020:00200000:3.0:1789760786.256395:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760786.256403:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760786.256439:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28553216, count: 4096 00000080:00200000:3.0:1789760786.256445:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28553216, bytes: 4096 00000080:00200000:3.0:1789760786.256452:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760786.256460:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760786.256462:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28553216, 28557312) 0 0 00000080:00200000:3.0:1789760786.256476:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760786.256747:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6971, 6971] 00020000:00200000:3.0:1789760786.256754:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28553216 chunk: [28553216, 28557312) 28557312, 4096 00020000:00200000:3.0:1789760786.256757:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760786.256773:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6971, 6971] 00000008:00010000:3.0:1789760786.256788:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760786.256793:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760786.256801:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28553216, 28557312) append 0 append lockpos 0 00000080:00200000:3.0:1789760786.256812:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28553216=0x1b3b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760786.256857:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760786.256859:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28557312 00000020:00010000:3.0:1789760786.256866:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760786.256869:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.256872:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.256875:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760786.256878:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760786.256884:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760786.256890:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760786.256937:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760786.257018:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae680 - object 0x240000400:5 offset 28553216<>28557312 00000100:00100000:2.0:1789760786.257081:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512756352:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760786.304511:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974290 gen 3 00000008:00010000:3.0:1789760786.304549:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:3.0:1789760786.304555:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.304565:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.304587:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:3.0:1789760786.304610:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512756352:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760786.304645:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6971, 6971] 00000080:00200000:0.0:1789760786.304658:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.304663:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.304675:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28557312 i_blocks 57351 00000020:00200000:0.0:1789760786.304679:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.304687:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.304714:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28557312, count: 4096 00000080:00200000:0.0:1789760786.304719:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28557312, bytes: 4096 00000080:00200000:0.0:1789760786.304727:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.304734:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.304737:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28557312, 28561408) 0 0 00000080:00200000:0.0:1789760786.304741:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.304754:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6972, 6972] 00020000:00200000:0.0:1789760786.304760:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28557312 chunk: [28557312, 28561408) 28561408, 4096 00020000:00200000:0.0:1789760786.304764:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.304778:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6972, 6972] 00000008:00010000:0.0:1789760786.304791:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.304796:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760786.304805:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28557312, 28561408) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.304815:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28557312=0x1b3c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.304856:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.304859:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28561408 00000020:00010000:0.0:1789760786.304865:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.304869:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.304872:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.304875:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.304877:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.304883:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.304888:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.304941:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000008:00100000:0.0:1789760786.305016:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148380 - object 0x240000400:5 offset 28557312<>28561408 00000100:00100000:0.0:1789760786.305042:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760786.305054:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512756480:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760786.345203:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974291 gen 3 00000008:00010000:0.0:1789760786.345240:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:0.0:1789760786.345248:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760786.345257:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760786.345281:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:0.0:1789760786.345305:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512756480:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760786.345404:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6972, 6972] 00000080:00200000:3.0:1789760786.345416:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760786.345422:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760786.345434:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28561408 i_blocks 57351 00000020:00200000:3.0:1789760786.345438:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760786.345445:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760786.345483:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28561408, count: 4096 00000080:00200000:3.0:1789760786.345489:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28561408, bytes: 4096 00000080:00200000:3.0:1789760786.345497:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760786.345505:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760786.345507:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28561408, 28565504) 0 0 00000080:00200000:3.0:1789760786.345511:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760786.345525:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6973, 6973] 00020000:00200000:3.0:1789760786.345532:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28561408 chunk: [28561408, 28565504) 28565504, 4096 00020000:00200000:3.0:1789760786.345536:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760786.345551:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6973, 6973] 00000008:00010000:3.0:1789760786.345565:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760786.345570:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760786.345580:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28561408, 28565504) append 0 append lockpos 0 00000080:00200000:3.0:1789760786.345590:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28561408=0x1b3d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760786.345635:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760786.345638:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28565504 00000020:00010000:3.0:1789760786.345644:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760786.345648:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.345650:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.345653:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760786.345656:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760786.345662:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760786.345668:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760786.345682:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760786.345759:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae680 - object 0x240000400:5 offset 28561408<>28565504 00000100:00100000:3.0:1789760786.345804:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760786.345817:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512756608:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760786.383032:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:3.0:1789760786.383043:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.383052:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.383073:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:3.0:1789760786.383104:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512756608:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760786.383192:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6973, 6973] 00000080:00200000:0.0:1789760786.383204:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.383209:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.383220:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28565504 i_blocks 57351 00000020:00200000:0.0:1789760786.383224:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.383232:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.383257:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28565504, count: 4096 00000080:00200000:0.0:1789760786.383262:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28565504, bytes: 4096 00000080:00200000:0.0:1789760786.383272:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.383279:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.383281:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28565504, 28569600) 0 0 00000080:00200000:0.0:1789760786.383285:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.383297:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6974, 6974] 00020000:00200000:0.0:1789760786.383304:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28565504 chunk: [28565504, 28569600) 28569600, 4096 00020000:00200000:0.0:1789760786.383308:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.383322:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6974, 6974] 00000008:00010000:0.0:1789760786.383335:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.383339:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760786.383347:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28565504, 28569600) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.383358:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28565504=0x1b3e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.383397:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.383399:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28569600 00000020:00010000:0.0:1789760786.383406:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.383409:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.383411:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.383414:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.383417:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.383423:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.383428:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.383440:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000008:00100000:0.0:1789760786.383511:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148380 - object 0x240000400:5 offset 28565504<>28569600 00000100:00100000:1.0:1789760786.383614:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512756736:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760786.430212:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974293 gen 3 00000008:00010000:1.0:1789760786.430269:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760786.430275:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.430283:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.430302:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760786.430324:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512756736:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760786.430418:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6974, 6974] 00000080:00200000:0.0:1789760786.430431:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.430437:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.430449:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28569600 i_blocks 57351 00000020:00200000:0.0:1789760786.430453:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.430461:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.430485:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28569600, count: 4096 00000080:00200000:0.0:1789760786.430492:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28569600, bytes: 4096 00000080:00200000:0.0:1789760786.430499:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.430506:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.430508:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28569600, 28573696) 0 0 00000080:00200000:0.0:1789760786.430511:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.430524:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6975, 6975] 00020000:00200000:0.0:1789760786.430530:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28569600 chunk: [28569600, 28573696) 28573696, 4096 00020000:00200000:0.0:1789760786.430535:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.430549:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6975, 6975] 00000008:00010000:0.0:1789760786.430563:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.430568:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760786.430577:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28569600, 28573696) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.430588:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28569600=0x1b3f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.430631:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.430634:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28573696 00000020:00010000:0.0:1789760786.430640:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.430643:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.430646:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.430649:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.430652:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.430658:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.430664:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.430677:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccc00 00000008:00100000:0.0:1789760786.430754:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b800 - object 0x240000400:5 offset 28569600<>28573696 00000100:00100000:1.0:1789760786.430837:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760786.430850:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512756864:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760786.468268:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974294 gen 3 00000008:00010000:1.0:1789760786.468304:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760786.468311:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.468320:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.468343:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760786.468368:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512756864:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760786.468465:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6975, 6975] 00000080:00200000:0.0:1789760786.468487:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.468492:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.468503:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28573696 i_blocks 57351 00000020:00200000:0.0:1789760786.468507:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.468514:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.468539:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28573696, count: 4096 00000080:00200000:0.0:1789760786.468545:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28573696, bytes: 4096 00000080:00200000:0.0:1789760786.468553:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.468560:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.468563:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28573696, 28577792) 0 0 00000080:00200000:0.0:1789760786.468566:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.468579:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6976, 6976] 00020000:00200000:0.0:1789760786.468586:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28573696 chunk: [28573696, 28577792) 28577792, 4096 00020000:00200000:0.0:1789760786.468590:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.468603:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6976, 6976] 00000008:00010000:0.0:1789760786.468615:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.468619:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760786.468628:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28573696, 28577792) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.468637:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28573696=0x1b40000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.468680:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.468683:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28577792 00000020:00010000:0.0:1789760786.468690:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.468693:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.468696:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.468699:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.468701:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.468708:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.468713:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.468727:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cce40 00000008:00100000:0.0:1789760786.468962:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b480 - object 0x240000400:5 offset 28573696<>28577792 00000100:00100000:1.0:1789760786.469226:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512756992:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760786.518633:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974295 gen 3 00000008:00010000:1.0:1789760786.518673:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760786.518680:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.518690:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.518714:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760786.518739:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512756992:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760786.519063:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6976, 6976] 00000080:00200000:0.0:1789760786.519078:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.519084:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.519107:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28577792 i_blocks 57351 00000020:00200000:0.0:1789760786.519110:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.519120:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.519274:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28577792, count: 4096 00000080:00200000:0.0:1789760786.519282:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28577792, bytes: 4096 00000080:00200000:0.0:1789760786.519291:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.519297:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.519300:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28577792, 28581888) 0 0 00000080:00200000:0.0:1789760786.519304:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.519317:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6977, 6977] 00020000:00200000:0.0:1789760786.519324:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28577792 chunk: [28577792, 28581888) 28581888, 4096 00020000:00200000:0.0:1789760786.519328:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.519345:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6977, 6977] 00000008:00010000:0.0:1789760786.519360:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.519365:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760786.519375:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28577792, 28581888) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.519387:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28577792=0x1b41000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.519436:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.519439:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28581888 00000020:00010000:0.0:1789760786.519445:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.519448:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.519451:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.519455:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.519457:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.519464:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.519470:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.519485:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cce40 00000008:00100000:0.0:1789760786.519568:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149180 - object 0x240000400:5 offset 28577792<>28581888 00000100:00100000:1.0:1789760786.519637:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512757120:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760786.556274:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974296 gen 3 00000008:00010000:1.0:1789760786.556311:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760786.556318:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.556328:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.556392:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760786.556416:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512757120:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760786.557335:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6977, 6977] 00000080:00200000:0.0:1789760786.557350:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.557356:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.557367:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28581888 i_blocks 57351 00000020:00200000:0.0:1789760786.557371:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.557379:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.557405:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28581888, count: 4096 00000080:00200000:0.0:1789760786.557412:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28581888, bytes: 4096 00000080:00200000:0.0:1789760786.557421:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.557428:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.557430:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28581888, 28585984) 0 0 00000080:00200000:0.0:1789760786.557434:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.557447:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6978, 6978] 00020000:00200000:0.0:1789760786.557455:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28581888 chunk: [28581888, 28585984) 28585984, 4096 00020000:00200000:0.0:1789760786.557458:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.557473:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6978, 6978] 00000008:00010000:0.0:1789760786.557486:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.557490:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760786.557499:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28581888, 28585984) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.557510:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28581888=0x1b42000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.557558:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.557560:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28585984 00000020:00010000:0.0:1789760786.557567:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.557571:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.557574:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.557577:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.557579:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.557586:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.557591:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.557610:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccc00 00000008:00100000:0.0:1789760786.557690:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148000 - object 0x240000400:5 offset 28581888<>28585984 00000100:00100000:1.0:1789760786.557769:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512757248:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760786.596973:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760786.596984:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.596993:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.597051:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760786.597075:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512757248:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760786.597221:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6978, 6978] 00000080:00200000:0.0:1789760786.597234:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.597239:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.597249:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28585984 i_blocks 57351 00000020:00200000:0.0:1789760786.597253:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.597260:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.597285:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28585984, count: 4096 00000080:00200000:0.0:1789760786.597291:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28585984, bytes: 4096 00000080:00200000:0.0:1789760786.597298:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.597305:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.597307:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28585984, 28590080) 0 0 00000080:00200000:0.0:1789760786.597310:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.597323:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6979, 6979] 00020000:00200000:0.0:1789760786.597329:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28585984 chunk: [28585984, 28590080) 28590080, 4096 00020000:00200000:0.0:1789760786.597333:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.597347:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6979, 6979] 00000008:00010000:0.0:1789760786.597358:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.597363:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760786.597371:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28585984, 28590080) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.597380:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28585984=0x1b43000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.597547:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.597550:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28590080 00000020:00010000:0.0:1789760786.597557:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.597560:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.597563:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.597566:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.597569:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.597575:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.597580:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.597594:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000008:00100000:0.0:1789760786.597670:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149c00 - object 0x240000400:5 offset 28585984<>28590080 00000100:00100000:1.0:1789760786.597734:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512757376:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760786.634331:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974298 gen 3 00000008:00010000:1.0:1789760786.634377:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760786.634384:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.634393:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.634412:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760786.634444:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512757376:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760786.634532:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6979, 6979] 00000080:00200000:3.0:1789760786.634545:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760786.634550:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760786.634561:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28590080 i_blocks 57351 00000020:00200000:3.0:1789760786.634564:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760786.634572:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760786.634597:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28590080, count: 4096 00000080:00200000:3.0:1789760786.634604:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28590080, bytes: 4096 00000080:00200000:3.0:1789760786.634611:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760786.634618:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760786.634621:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28590080, 28594176) 0 0 00000080:00200000:3.0:1789760786.634624:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760786.634637:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6980, 6980] 00020000:00200000:3.0:1789760786.634644:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28590080 chunk: [28590080, 28594176) 28594176, 4096 00020000:00200000:3.0:1789760786.634648:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760786.634661:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6980, 6980] 00000008:00010000:3.0:1789760786.634673:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760786.634678:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760786.634687:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28590080, 28594176) append 0 append lockpos 0 00000080:00200000:3.0:1789760786.634697:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28590080=0x1b44000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760786.634740:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760786.634742:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28594176 00000020:00010000:3.0:1789760786.634749:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760786.634752:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.634754:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.634757:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760786.634760:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760786.634766:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760786.634772:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760786.634786:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760786.634857:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 28590080<>28594176 00000100:00100000:3.0:1789760786.634930:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760786.634943:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512757504:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760786.681010:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974299 gen 3 00000008:00010000:3.0:1789760786.681048:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:3.0:1789760786.681054:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.681063:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.681096:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:3.0:1789760786.681121:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512757504:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760786.681223:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6980, 6980] 00000080:00200000:0.0:1789760786.681237:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.681243:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.681256:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28594176 i_blocks 57351 00000020:00200000:0.0:1789760786.681260:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.681269:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.681294:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28594176, count: 4096 00000080:00200000:0.0:1789760786.681301:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28594176, bytes: 4096 00000080:00200000:0.0:1789760786.681309:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.681317:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.681318:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28594176, 28598272) 0 0 00000080:00200000:0.0:1789760786.681322:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.681335:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6981, 6981] 00020000:00200000:0.0:1789760786.681343:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28594176 chunk: [28594176, 28598272) 28598272, 4096 00020000:00200000:0.0:1789760786.681346:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.681361:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6981, 6981] 00000008:00010000:0.0:1789760786.681376:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.681380:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760786.681389:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28594176, 28598272) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.681399:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28594176=0x1b45000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.681442:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.681446:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28598272 00000020:00010000:0.0:1789760786.681452:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.681455:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.681458:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.681462:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.681465:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.681472:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.681487:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.681501:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc180 00000008:00100000:0.0:1789760786.681709:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514bb80 - object 0x240000400:5 offset 28594176<>28598272 00000100:00100000:1.0:1789760786.682188:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512757632:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760786.731669:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974300 gen 3 00000008:00010000:1.0:1789760786.731708:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760786.731714:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.731723:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.731746:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760786.731770:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512757632:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760786.731995:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6981, 6981] 00000080:00200000:0.0:1789760786.732008:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.732014:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.732026:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28598272 i_blocks 57351 00000020:00200000:0.0:1789760786.732029:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.732037:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.732063:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28598272, count: 4096 00000080:00200000:0.0:1789760786.732070:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28598272, bytes: 4096 00000080:00200000:0.0:1789760786.732077:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.732093:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.732096:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28598272, 28602368) 0 0 00000080:00200000:0.0:1789760786.732100:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.732113:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6982, 6982] 00020000:00200000:0.0:1789760786.732120:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28598272 chunk: [28598272, 28602368) 28602368, 4096 00020000:00200000:0.0:1789760786.732124:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.732139:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6982, 6982] 00000008:00010000:0.0:1789760786.732153:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.732158:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760786.732166:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28598272, 28602368) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.732176:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28598272=0x1b46000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.732219:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.732222:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28602368 00000020:00010000:0.0:1789760786.732229:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.732232:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.732235:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.732238:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.732241:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.732247:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.732252:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.732268:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccd80 00000008:00100000:0.0:1789760786.732345:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149f80 - object 0x240000400:5 offset 28598272<>28602368 00000100:00100000:1.0:1789760786.732409:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512757760:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760786.796732:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760786.796742:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760786.796752:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760786.796830:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000080:00200000:0.0:1789760786.797204:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6982, 6982] 00000080:00200000:0.0:1789760786.797218:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.797224:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.797235:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28602368 i_blocks 57351 00000020:00200000:0.0:1789760786.797238:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.797247:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.797270:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28602368, count: 4096 00000080:00200000:0.0:1789760786.797277:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28602368, bytes: 4096 00000080:00200000:0.0:1789760786.797284:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.797292:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.797294:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28602368, 28606464) 0 0 00000080:00200000:0.0:1789760786.797298:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.797310:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6983, 6983] 00020000:00200000:0.0:1789760786.797317:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28602368 chunk: [28602368, 28606464) 28606464, 4096 00020000:00200000:0.0:1789760786.797320:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.797335:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6983, 6983] 00000008:00010000:0.0:1789760786.797347:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.797352:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760786.797361:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28602368, 28606464) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.797371:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28602368=0x1b47000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.797416:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.797418:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28606464 00000020:00010000:0.0:1789760786.797424:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.797427:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.797430:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.797433:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.797435:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.797442:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.797448:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.797465:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760786.797731:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512757760:192.168.203.156@tcp:4:dd.0 00000008:00100000:0.0:1789760786.797840:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149500 - object 0x240000400:5 offset 28602368<>28606464 00000100:00100000:1.0:1789760786.797965:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760786.798229:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512757888:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760786.836304:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974301 gen 3 00000100:00100000:0.0:1789760786.836335:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005149500 x1876699512757888/t4294974302(4294974302) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760802 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760786.836374:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:0.0:1789760786.836379:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760786.836388:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760786.836454:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:0.0:1789760786.836474:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512757888:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760786.836599:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6983, 6983] 00000080:00200000:3.0:1789760786.836612:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760786.836617:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760786.836628:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28606464 i_blocks 57351 00000020:00200000:3.0:1789760786.836631:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760786.836639:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760786.836741:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28606464, count: 4096 00000080:00200000:3.0:1789760786.836748:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28606464, bytes: 4096 00000080:00200000:3.0:1789760786.836817:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760786.836825:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760786.836827:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28606464, 28610560) 0 0 00000080:00200000:3.0:1789760786.836831:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760786.836843:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6984, 6984] 00020000:00200000:3.0:1789760786.836851:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28606464 chunk: [28606464, 28610560) 28610560, 4096 00020000:00200000:3.0:1789760786.836855:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760786.836948:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6984, 6984] 00000008:00010000:3.0:1789760786.836963:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760786.836968:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760786.836978:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28606464, 28610560) append 0 append lockpos 0 00000080:00200000:3.0:1789760786.836988:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28606464=0x1b48000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760786.837117:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760786.837120:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28610560 00000020:00010000:3.0:1789760786.837127:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760786.837130:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.837133:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.837136:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760786.837139:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760786.837145:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760786.837150:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760786.837166:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760786.837241:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 28606464<>28610560 00000100:00100000:2.0:1789760786.837294:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760786.837357:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512758016:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760786.865984:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974303 gen 3 00000008:00010000:3.0:1789760786.866034:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:3.0:1789760786.866042:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.866050:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.866072:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:3.0:1789760786.866103:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512758016:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760786.866119:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6984, 6984] 00000080:00200000:0.0:1789760786.866132:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.866137:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.866148:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28610560 i_blocks 57351 00000020:00200000:0.0:1789760786.866153:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.866160:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.866186:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28610560, count: 4096 00000080:00200000:0.0:1789760786.866192:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28610560, bytes: 4096 00000080:00200000:0.0:1789760786.866200:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.866207:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.866209:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28610560, 28614656) 0 0 00000080:00200000:0.0:1789760786.866214:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.866228:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6985, 6985] 00020000:00200000:0.0:1789760786.866235:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28610560 chunk: [28610560, 28614656) 28614656, 4096 00020000:00200000:0.0:1789760786.866239:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.866253:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6985, 6985] 00000008:00010000:0.0:1789760786.866267:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.866272:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760786.866281:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28610560, 28614656) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.866293:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28610560=0x1b49000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.866339:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.866342:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28614656 00000020:00010000:0.0:1789760786.866349:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.866353:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.866356:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.866359:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.866361:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.866367:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.866374:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.866388:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc780 00000008:00100000:0.0:1789760786.866462:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149f80 - object 0x240000400:5 offset 28610560<>28614656 00000100:00100000:0.0:1789760786.866507:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512758144:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760786.898776:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:0.0:1789760786.898787:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760786.898796:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760786.898813:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:0.0:1789760786.898896:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512758144:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760786.899153:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6985, 6985] 00000080:00200000:3.0:1789760786.899165:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760786.899170:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760786.899180:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28614656 i_blocks 57351 00000020:00200000:3.0:1789760786.899183:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760786.899191:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760786.899216:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28614656, count: 4096 00000080:00200000:3.0:1789760786.899221:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28614656, bytes: 4096 00000080:00200000:3.0:1789760786.899228:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760786.899235:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760786.899237:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28614656, 28618752) 0 0 00000080:00200000:3.0:1789760786.899241:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760786.899253:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6986, 6986] 00020000:00200000:3.0:1789760786.899258:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28614656 chunk: [28614656, 28618752) 28618752, 4096 00020000:00200000:3.0:1789760786.899262:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760786.899274:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6986, 6986] 00000008:00010000:3.0:1789760786.899287:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760786.899291:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760786.899298:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28614656, 28618752) append 0 append lockpos 0 00000080:00200000:3.0:1789760786.899308:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28614656=0x1b4a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760786.899346:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760786.899348:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28618752 00000020:00010000:3.0:1789760786.899353:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760786.899356:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.899359:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760786.899362:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760786.899365:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760786.899371:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760786.899375:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760786.899387:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760786.899459:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 28614656<>28618752 00000100:00100000:2.0:1789760786.899696:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760786.903666:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512758272:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760786.968954:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974305 gen 3 00000008:00010000:3.0:1789760786.969014:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:3.0:1789760786.969022:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760786.969041:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760786.969053:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:3.0:1789760786.969078:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512758272:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760786.969421:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6986, 6986] 00000080:00200000:0.0:1789760786.969435:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760786.969440:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760786.969453:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28618752 i_blocks 57351 00000020:00200000:0.0:1789760786.969456:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760786.969464:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760786.969634:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28618752, count: 4096 00000080:00200000:0.0:1789760786.969641:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28618752, bytes: 4096 00000080:00200000:0.0:1789760786.969650:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760786.969657:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760786.969660:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28618752, 28622848) 0 0 00000080:00200000:0.0:1789760786.969664:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760786.969676:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6987, 6987] 00020000:00200000:0.0:1789760786.969683:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28618752 chunk: [28618752, 28622848) 28622848, 4096 00020000:00200000:0.0:1789760786.969687:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760786.969702:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6987, 6987] 00000008:00010000:0.0:1789760786.969715:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760786.969720:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760786.969729:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28618752, 28622848) append 0 append lockpos 0 00000080:00200000:0.0:1789760786.969740:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28618752=0x1b4b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760786.969788:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760786.969791:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28622848 00000020:00010000:0.0:1789760786.969797:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760786.969800:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.969802:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760786.969806:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760786.969809:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760786.969817:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760786.969823:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760786.969839:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc780 00000008:00100000:0.0:1789760786.969964:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b100 - object 0x240000400:5 offset 28618752<>28622848 00000100:00100000:0.0:1789760786.970002:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512758400:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760787.034957:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974306 gen 3 00000008:00010000:0.0:1789760787.034995:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:0.0:1789760787.035001:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760787.035012:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760787.035160:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:0.0:1789760787.035184:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512758400:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760787.035269:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6987, 6987] 00000080:00200000:3.0:1789760787.035282:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760787.035287:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760787.035298:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28622848 i_blocks 57351 00000020:00200000:3.0:1789760787.035301:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760787.035309:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760787.035333:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28622848, count: 4096 00000080:00200000:3.0:1789760787.035338:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28622848, bytes: 4096 00000080:00200000:3.0:1789760787.035345:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760787.035353:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760787.035354:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28622848, 28626944) 0 0 00000080:00200000:3.0:1789760787.035358:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760787.035371:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6988, 6988] 00020000:00200000:3.0:1789760787.035377:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28622848 chunk: [28622848, 28626944) 28626944, 4096 00020000:00200000:3.0:1789760787.035380:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760787.035394:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6988, 6988] 00000008:00010000:3.0:1789760787.035406:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760787.035410:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760787.035419:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28622848, 28626944) append 0 append lockpos 0 00000080:00200000:3.0:1789760787.035430:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28622848=0x1b4c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760787.035471:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760787.035473:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28626944 00000020:00010000:3.0:1789760787.035479:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760787.035482:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.035485:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.035488:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760787.035491:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760787.035497:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760787.035502:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760787.036006:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760787.036144:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 28622848<>28626944 00000100:00100000:2.0:1789760787.036203:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760787.043518:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512758528:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760787.089735:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974307 gen 3 00000008:00010000:3.0:1789760787.089773:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:3.0:1789760787.089780:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.089789:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.089811:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:3.0:1789760787.089834:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512758528:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760787.089931:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6988, 6988] 00000080:00200000:0.0:1789760787.089943:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760787.089949:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760787.089961:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28626944 i_blocks 57351 00000020:00200000:0.0:1789760787.089965:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760787.089974:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760787.090004:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28626944, count: 4096 00000080:00200000:0.0:1789760787.090010:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28626944, bytes: 4096 00000080:00200000:0.0:1789760787.090018:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760787.090025:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760787.090027:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28626944, 28631040) 0 0 00000080:00200000:0.0:1789760787.090043:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760787.090057:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6989, 6989] 00020000:00200000:0.0:1789760787.090063:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28626944 chunk: [28626944, 28631040) 28631040, 4096 00020000:00200000:0.0:1789760787.090066:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760787.090092:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6989, 6989] 00000008:00010000:0.0:1789760787.090108:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760787.090113:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760787.090125:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28626944, 28631040) append 0 append lockpos 0 00000080:00200000:0.0:1789760787.090138:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28626944=0x1b4d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760787.090266:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760787.090270:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28631040 00000020:00010000:0.0:1789760787.090277:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760787.090281:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.090284:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.090287:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760787.090290:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760787.090296:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760787.090302:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760787.090317:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc780 00000008:00100000:0.0:1789760787.090395:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b100 - object 0x240000400:5 offset 28626944<>28631040 00000100:00100000:0.0:1789760787.090441:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512758656:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760787.141115:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974308 gen 3 00000008:00010000:0.0:1789760787.141158:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:0.0:1789760787.141164:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760787.141173:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760787.141210:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:0.0:1789760787.141235:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512758656:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760787.141394:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6989, 6989] 00000080:00200000:3.0:1789760787.141407:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760787.141412:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760787.141424:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28631040 i_blocks 57351 00000020:00200000:3.0:1789760787.141428:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760787.141437:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760787.141465:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28631040, count: 4096 00000080:00200000:3.0:1789760787.141472:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28631040, bytes: 4096 00000080:00200000:3.0:1789760787.141483:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760787.141491:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760787.141493:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28631040, 28635136) 0 0 00000080:00200000:3.0:1789760787.141497:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760787.141521:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6990, 6990] 00020000:00200000:3.0:1789760787.141529:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28631040 chunk: [28631040, 28635136) 28635136, 4096 00020000:00200000:3.0:1789760787.141534:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760787.141551:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6990, 6990] 00000008:00010000:3.0:1789760787.141566:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760787.141570:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760787.141580:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28631040, 28635136) append 0 append lockpos 0 00000080:00200000:3.0:1789760787.141591:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28631040=0x1b4e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760787.141637:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760787.141640:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28635136 00000020:00010000:3.0:1789760787.141647:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760787.141650:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.141653:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.141656:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760787.141659:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760787.141665:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760787.141671:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760787.141686:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760787.141774:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 28631040<>28635136 00000100:00100000:3.0:1789760787.141823:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512758784:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760787.212498:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974309 gen 3 00000008:00010000:3.0:1789760787.212536:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:3.0:1789760787.212541:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.212550:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.212570:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:3.0:1789760787.212591:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512758784:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760787.212713:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6990, 6990] 00000080:00200000:0.0:1789760787.212725:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760787.212731:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760787.212741:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28635136 i_blocks 57351 00000020:00200000:0.0:1789760787.212744:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760787.212752:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760787.212775:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28635136, count: 4096 00000080:00200000:0.0:1789760787.212781:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28635136, bytes: 4096 00000080:00200000:0.0:1789760787.212788:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760787.212795:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760787.212797:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28635136, 28639232) 0 0 00000080:00200000:0.0:1789760787.212800:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760787.212812:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6991, 6991] 00020000:00200000:0.0:1789760787.212817:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28635136 chunk: [28635136, 28639232) 28639232, 4096 00020000:00200000:0.0:1789760787.212821:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760787.212835:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6991, 6991] 00000008:00010000:0.0:1789760787.212847:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760787.212852:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760787.212861:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28635136, 28639232) append 0 append lockpos 0 00000080:00200000:0.0:1789760787.212870:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28635136=0x1b4f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760787.212941:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760787.212944:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28639232 00000020:00010000:0.0:1789760787.212951:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760787.212954:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.212957:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.212960:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760787.212963:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760787.212969:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760787.212973:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760787.212987:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc780 00000008:00100000:0.0:1789760787.213076:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b100 - object 0x240000400:5 offset 28635136<>28639232 00000100:00100000:0.0:1789760787.213115:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760787.213126:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512758912:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760787.229078:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000008:00010000:0.0:1789760787.258055:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:0.0:1789760787.258066:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760787.258075:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760787.258139:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:0.0:1789760787.258160:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512758912:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760787.258328:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6991, 6991] 00000080:00200000:3.0:1789760787.258341:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760787.258345:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760787.258356:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28639232 i_blocks 57351 00000020:00200000:3.0:1789760787.258359:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760787.258366:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760787.258390:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28639232, count: 4096 00000080:00200000:3.0:1789760787.258396:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28639232, bytes: 4096 00000080:00200000:3.0:1789760787.258403:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760787.258410:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760787.258412:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28639232, 28643328) 0 0 00000080:00200000:3.0:1789760787.258416:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760787.258427:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6992, 6992] 00020000:00200000:3.0:1789760787.258433:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28639232 chunk: [28639232, 28643328) 28643328, 4096 00020000:00200000:3.0:1789760787.258437:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760787.258450:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6992, 6992] 00000008:00010000:3.0:1789760787.258462:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760787.258466:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760787.258535:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28639232, 28643328) append 0 append lockpos 0 00000080:00200000:3.0:1789760787.258547:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28639232=0x1b50000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760787.258588:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760787.258590:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28643328 00000020:00010000:3.0:1789760787.258596:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760787.258600:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.258603:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.258606:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760787.258608:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760787.258614:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760787.258619:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760787.258632:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760787.258698:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 28639232<>28643328 00000100:00100000:2.0:1789760787.258761:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512759040:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760787.298369:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974311 gen 3 00000008:00010000:3.0:1789760787.298412:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:3.0:1789760787.298419:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.298427:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.298445:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:3.0:1789760787.298466:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512759040:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760787.299409:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6992, 6992] 00000080:00200000:0.0:1789760787.299423:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760787.299428:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760787.299440:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28643328 i_blocks 57351 00000020:00200000:0.0:1789760787.299444:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760787.299453:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760787.299489:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28643328, count: 4096 00000080:00200000:0.0:1789760787.299497:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28643328, bytes: 4096 00000080:00200000:0.0:1789760787.299505:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760787.299512:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760787.299514:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28643328, 28647424) 0 0 00000080:00200000:0.0:1789760787.299517:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760787.299531:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6993, 6993] 00020000:00200000:0.0:1789760787.299540:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28643328 chunk: [28643328, 28647424) 28647424, 4096 00020000:00200000:0.0:1789760787.299543:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760787.299560:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6993, 6993] 00000008:00010000:0.0:1789760787.299573:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760787.299578:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760787.299587:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28643328, 28647424) append 0 append lockpos 0 00000080:00200000:0.0:1789760787.299597:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28643328=0x1b51000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760787.299644:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760787.299647:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28647424 00000020:00010000:0.0:1789760787.299653:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760787.299656:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.299659:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.299662:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760787.299665:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760787.299672:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760787.299678:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760787.299694:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc780 00000008:00100000:0.0:1789760787.299773:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b100 - object 0x240000400:5 offset 28643328<>28647424 00000100:00100000:1.0:1789760787.299924:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760787.299942:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512759168:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760787.332838:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974312 gen 3 00000008:00010000:1.0:1789760787.332875:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760787.332883:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.332892:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.332934:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760787.332955:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512759168:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760787.333127:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6993, 6993] 00000080:00200000:0.0:1789760787.333139:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760787.333145:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760787.333156:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28647424 i_blocks 57351 00000020:00200000:0.0:1789760787.333159:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760787.333167:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760787.333193:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28647424, count: 4096 00000080:00200000:0.0:1789760787.333199:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28647424, bytes: 4096 00000080:00200000:0.0:1789760787.333207:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760787.333214:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760787.333216:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28647424, 28651520) 0 0 00000080:00200000:0.0:1789760787.333220:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760787.333232:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6994, 6994] 00020000:00200000:0.0:1789760787.333240:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28647424 chunk: [28647424, 28651520) 28651520, 4096 00020000:00200000:0.0:1789760787.333244:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760787.333257:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6994, 6994] 00000008:00010000:0.0:1789760787.333270:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760787.333275:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760787.333283:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28647424, 28651520) append 0 append lockpos 0 00000080:00200000:0.0:1789760787.333295:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28647424=0x1b52000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760787.333337:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760787.333340:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28651520 00000020:00010000:0.0:1789760787.333346:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760787.333350:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.333353:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.333356:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760787.333359:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760787.333365:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760787.333370:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760787.333384:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000008:00100000:0.0:1789760787.333462:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149f80 - object 0x240000400:5 offset 28647424<>28651520 00000100:00100000:0.0:1789760787.333495:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760787.333507:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512759296:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760787.370371:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974313 gen 3 00000008:00010000:0.0:1789760787.370406:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:0.0:1789760787.370413:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760787.370423:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760787.370438:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:0.0:1789760787.370462:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512759296:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760787.370534:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6994, 6994] 00000080:00200000:3.0:1789760787.370547:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760787.370552:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760787.370564:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28651520 i_blocks 57351 00000020:00200000:3.0:1789760787.370567:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760787.370575:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760787.370730:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28651520, count: 4096 00000080:00200000:3.0:1789760787.370738:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28651520, bytes: 4096 00000080:00200000:3.0:1789760787.370747:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760787.370754:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760787.370756:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28651520, 28655616) 0 0 00000080:00200000:3.0:1789760787.370760:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760787.370772:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6995, 6995] 00020000:00200000:3.0:1789760787.370780:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28651520 chunk: [28651520, 28655616) 28655616, 4096 00020000:00200000:3.0:1789760787.370784:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760787.370798:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6995, 6995] 00000008:00010000:3.0:1789760787.370813:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760787.370817:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760787.370827:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28651520, 28655616) append 0 append lockpos 0 00000080:00200000:3.0:1789760787.370838:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28651520=0x1b53000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760787.370882:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760787.370885:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28655616 00000020:00010000:3.0:1789760787.370891:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760787.370895:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.370897:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.370988:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760787.370992:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760787.370999:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760787.371004:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760787.371019:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760787.371105:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 28651520<>28655616 00000100:00100000:2.0:1789760787.371288:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760787.371359:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512759424:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760787.422319:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:3.0:1789760787.422331:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.422342:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.422363:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:3.0:1789760787.422385:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512759424:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760787.422467:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6995, 6995] 00000080:00200000:2.0:1789760787.422538:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.422544:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.422556:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28655616 i_blocks 57351 00000020:00200000:2.0:1789760787.422560:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.422567:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.422590:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28655616, count: 4096 00000080:00200000:2.0:1789760787.422596:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28655616, bytes: 4096 00000080:00200000:2.0:1789760787.422604:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.422611:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.422613:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28655616, 28659712) 0 0 00000080:00200000:2.0:1789760787.422617:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.422628:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6996, 6996] 00020000:00200000:2.0:1789760787.422634:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28655616 chunk: [28655616, 28659712) 28659712, 4096 00020000:00200000:2.0:1789760787.422638:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.422650:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6996, 6996] 00000008:00010000:2.0:1789760787.422662:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.422667:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760787.422675:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28655616, 28659712) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.422686:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28655616=0x1b54000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.422725:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.422727:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28659712 00000020:00010000:2.0:1789760787.422733:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.422737:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.422739:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.422742:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.422745:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.422750:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.422755:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.422767:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760787.422834:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 28655616<>28659712 00000100:00100000:3.0:1789760787.422879:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512759552:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760787.467128:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974315 gen 3 00000008:00010000:3.0:1789760787.467183:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760787.467189:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.467197:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.467213:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760787.467238:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512759552:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760787.467392:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6996, 6996] 00000080:00200000:2.0:1789760787.467405:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.467410:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.467423:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28659712 i_blocks 57351 00000020:00200000:2.0:1789760787.467426:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.467434:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.467460:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28659712, count: 4096 00000080:00200000:2.0:1789760787.467465:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28659712, bytes: 4096 00000080:00200000:2.0:1789760787.467479:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.467486:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.467488:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28659712, 28663808) 0 0 00000080:00200000:2.0:1789760787.467492:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.467506:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6997, 6997] 00020000:00200000:2.0:1789760787.467513:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28659712 chunk: [28659712, 28663808) 28663808, 4096 00020000:00200000:2.0:1789760787.467516:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.467530:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6997, 6997] 00000008:00010000:2.0:1789760787.467542:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.467547:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760787.467555:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28659712, 28663808) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.467566:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28659712=0x1b55000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.467625:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.467628:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28663808 00000020:00010000:2.0:1789760787.467635:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.467639:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.467641:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.467644:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.467647:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.467653:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.467659:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.467673:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760787.467754:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 28659712<>28663808 00000100:00100000:2.0:1789760787.467805:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512759680:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760787.503746:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974316 gen 3 00000008:00010000:3.0:1789760787.503798:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760787.503804:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.503813:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.503829:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760787.503853:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512759680:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760787.503948:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6997, 6997] 00000080:00200000:2.0:1789760787.503961:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.503966:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.503977:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28663808 i_blocks 57351 00000020:00200000:2.0:1789760787.503981:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.503988:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.504013:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28663808, count: 4096 00000080:00200000:2.0:1789760787.504019:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28663808, bytes: 4096 00000080:00200000:2.0:1789760787.504027:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.504034:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.504036:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28663808, 28667904) 0 0 00000080:00200000:2.0:1789760787.504040:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.504052:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6998, 6998] 00020000:00200000:2.0:1789760787.504059:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28663808 chunk: [28663808, 28667904) 28667904, 4096 00020000:00200000:2.0:1789760787.504063:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.504077:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6998, 6998] 00000008:00010000:2.0:1789760787.504100:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.504106:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760787.504114:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28663808, 28667904) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.504126:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28663808=0x1b56000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.504180:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.504183:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28667904 00000020:00010000:2.0:1789760787.504189:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.504192:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.504196:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.504199:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.504201:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.504208:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.504214:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.504227:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760787.504299:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 28663808<>28667904 00000100:00100000:3.0:1789760787.504511:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512759808:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760787.539187:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974317 gen 3 00000008:00010000:3.0:1789760787.539223:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760787.539229:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.539237:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.539255:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760787.539278:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512759808:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760787.539361:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6998, 6998] 00000080:00200000:2.0:1789760787.539373:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.539378:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.539391:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28667904 i_blocks 57351 00000020:00200000:2.0:1789760787.539394:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.539401:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.539424:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28667904, count: 4096 00000080:00200000:2.0:1789760787.539430:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28667904, bytes: 4096 00000080:00200000:2.0:1789760787.539436:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.539443:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.539445:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28667904, 28672000) 0 0 00000080:00200000:2.0:1789760787.539449:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.539462:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [6999, 6999] 00020000:00200000:2.0:1789760787.539468:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28667904 chunk: [28667904, 28672000) 28672000, 4096 00020000:00200000:2.0:1789760787.539480:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.539495:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [6999, 6999] 00000008:00010000:2.0:1789760787.539509:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.539514:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760787.539522:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28667904, 28672000) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.539532:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28667904=0x1b57000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.539572:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.539575:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28672000 00000020:00010000:2.0:1789760787.539581:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.539584:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.539586:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.539589:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.539592:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.539597:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.539602:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.539615:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760787.539686:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 28667904<>28672000 00000100:00100000:3.0:1789760787.539882:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512759936:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760787.572044:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974318 gen 3 00000008:00010000:3.0:1789760787.572087:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760787.572093:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.572101:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.572117:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760787.572139:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512759936:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760787.572219:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [6999, 6999] 00000080:00200000:2.0:1789760787.572231:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.572235:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.572246:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28672000 i_blocks 57351 00000020:00200000:2.0:1789760787.572248:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.572256:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.572281:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28672000, count: 4096 00000080:00200000:2.0:1789760787.572287:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28672000, bytes: 4096 00000080:00200000:2.0:1789760787.572293:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.572300:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.572302:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28672000, 28676096) 0 0 00000080:00200000:2.0:1789760787.572306:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.572319:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7000, 7000] 00020000:00200000:2.0:1789760787.572326:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28672000 chunk: [28672000, 28676096) 28676096, 4096 00020000:00200000:2.0:1789760787.572329:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.572343:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7000, 7000] 00000008:00010000:2.0:1789760787.572356:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.572360:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760787.572369:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28672000, 28676096) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.572380:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28672000=0x1b58000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.572419:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.572421:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28676096 00000020:00010000:2.0:1789760787.572427:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.572431:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.572433:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.572437:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.572440:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.572445:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.572450:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.572462:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760787.572543:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 28672000<>28676096 00000100:00100000:3.0:1789760787.572593:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512760064:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760787.598050:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760787.598061:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.598069:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.598127:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760787.598151:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512760064:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760787.598208:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7000, 7000] 00000080:00200000:2.0:1789760787.598219:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.598225:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.598235:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28676096 i_blocks 57351 00000020:00200000:2.0:1789760787.598239:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.598245:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.598270:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28676096, count: 4096 00000080:00200000:2.0:1789760787.598276:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28676096, bytes: 4096 00000080:00200000:2.0:1789760787.598283:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.598289:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.598291:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28676096, 28680192) 0 0 00000080:00200000:2.0:1789760787.598295:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.598307:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7001, 7001] 00020000:00200000:2.0:1789760787.598313:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28676096 chunk: [28676096, 28680192) 28680192, 4096 00020000:00200000:2.0:1789760787.598317:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.598332:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7001, 7001] 00000008:00010000:2.0:1789760787.598345:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.598350:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760787.598357:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28676096, 28680192) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.598367:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28676096=0x1b59000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.598408:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.598410:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28680192 00000020:00010000:2.0:1789760787.598416:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.598419:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.598422:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.598424:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.598427:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.598432:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.598437:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.598450:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760787.598633:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 28676096<>28680192 00000100:00100000:3.0:1789760787.598708:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512760192:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760787.638163:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974320 gen 3 00000008:00010000:3.0:1789760787.638210:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760787.638216:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.638224:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.638762:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760787.638789:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512760192:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760787.640097:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7001, 7001] 00000080:00200000:2.0:1789760787.640115:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.640176:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.640191:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28680192 i_blocks 57351 00000020:00200000:2.0:1789760787.640195:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.640205:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.640232:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28680192, count: 4096 00000080:00200000:2.0:1789760787.640238:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28680192, bytes: 4096 00000080:00200000:2.0:1789760787.640249:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.640256:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.640258:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28680192, 28684288) 0 0 00000080:00200000:2.0:1789760787.640263:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.640277:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7002, 7002] 00020000:00200000:2.0:1789760787.640284:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28680192 chunk: [28680192, 28684288) 28684288, 4096 00020000:00200000:2.0:1789760787.640287:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.640303:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7002, 7002] 00000008:00010000:2.0:1789760787.640318:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.640322:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760787.640330:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28680192, 28684288) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.640341:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28680192=0x1b5a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.640393:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.640396:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28684288 00000020:00010000:2.0:1789760787.640403:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.640406:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.640409:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.640412:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.640415:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.640422:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.640427:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.640447:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760787.640788:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 28680192<>28684288 00000100:00100000:3.0:1789760787.640855:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512760320:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760787.707557:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760787.707569:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760787.707577:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760787.707604:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760787.707629:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512760320:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760787.707651:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7002, 7002] 00000080:00200000:2.0:1789760787.707665:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.707671:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.707683:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28684288 i_blocks 57351 00000020:00200000:2.0:1789760787.707686:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.707694:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.707718:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28684288, count: 4096 00000080:00200000:2.0:1789760787.707724:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28684288, bytes: 4096 00000080:00200000:2.0:1789760787.707733:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.707739:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.707741:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28684288, 28688384) 0 0 00000080:00200000:2.0:1789760787.707745:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.707757:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7003, 7003] 00020000:00200000:2.0:1789760787.707764:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28684288 chunk: [28684288, 28688384) 28688384, 4096 00020000:00200000:2.0:1789760787.707768:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.707782:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7003, 7003] 00000008:00010000:2.0:1789760787.707804:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.707809:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760787.707818:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28684288, 28688384) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.707828:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28684288=0x1b5b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.707867:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.707869:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28688384 00000020:00010000:2.0:1789760787.707875:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.707878:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.707881:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.707884:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.707886:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.707892:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.707897:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.707929:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760787.708003:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 28684288<>28688384 00000100:00100000:3.0:1789760787.708064:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512760448:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760787.748721:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974322 gen 3 00000008:00010000:2.0:1789760787.748780:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:2.0:1789760787.748972:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760787.748982:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760787.748997:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:2.0:1789760787.749020:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512760448:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760787.749155:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7003, 7003] 00000080:00200000:3.0:1789760787.749169:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760787.749174:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760787.749186:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28688384 i_blocks 57351 00000020:00200000:3.0:1789760787.749189:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760787.749197:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760787.749222:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28688384, count: 4096 00000080:00200000:3.0:1789760787.749229:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28688384, bytes: 4096 00000080:00200000:3.0:1789760787.749237:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760787.749244:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760787.749247:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28688384, 28692480) 0 0 00000080:00200000:3.0:1789760787.749250:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760787.749264:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7004, 7004] 00020000:00200000:3.0:1789760787.749269:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28688384 chunk: [28688384, 28692480) 28692480, 4096 00020000:00200000:3.0:1789760787.749273:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760787.749290:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7004, 7004] 00000008:00010000:3.0:1789760787.749302:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760787.749307:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760787.749316:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28688384, 28692480) append 0 append lockpos 0 00000080:00200000:3.0:1789760787.749328:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28688384=0x1b5c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760787.749371:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760787.749374:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28692480 00000020:00010000:3.0:1789760787.749381:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760787.749384:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.749387:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760787.749389:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760787.749392:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760787.749398:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760787.749404:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760787.749418:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760787.749506:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 28688384<>28692480 00000100:00100000:2.0:1789760787.749565:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512760576:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760787.804837:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974323 gen 3 00000008:00010000:2.0:1789760787.804876:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760787.804882:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760787.804891:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760787.804941:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760787.804964:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512760576:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760787.805140:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7004, 7004] 00000080:00200000:0.0:1789760787.805153:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760787.805158:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760787.805171:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28692480 i_blocks 57351 00000020:00200000:0.0:1789760787.805174:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760787.805182:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760787.805206:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28692480, count: 4096 00000080:00200000:0.0:1789760787.805211:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28692480, bytes: 4096 00000080:00200000:0.0:1789760787.805219:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760787.805226:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760787.805228:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28692480, 28696576) 0 0 00000080:00200000:0.0:1789760787.805232:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760787.805245:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7005, 7005] 00020000:00200000:0.0:1789760787.805251:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28692480 chunk: [28692480, 28696576) 28696576, 4096 00020000:00200000:0.0:1789760787.805255:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760787.805269:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7005, 7005] 00000008:00010000:0.0:1789760787.805281:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760787.805286:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760787.805294:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28692480, 28696576) append 0 append lockpos 0 00000080:00200000:0.0:1789760787.805305:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28692480=0x1b5d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760787.805349:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760787.805351:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28696576 00000020:00010000:0.0:1789760787.805358:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760787.805361:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.805364:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.805367:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760787.805369:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760787.805377:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760787.805382:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760787.805394:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000008:00100000:0.0:1789760787.805465:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149f80 - object 0x240000400:5 offset 28692480<>28696576 00000100:00100000:1.0:1789760787.805601:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760787.805616:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512760704:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760787.834067:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760787.834087:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.834097:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.834116:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760787.834140:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512760704:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760787.834230:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7005, 7005] 00000080:00200000:0.0:1789760787.834242:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760787.834247:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760787.834259:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28696576 i_blocks 57351 00000020:00200000:0.0:1789760787.834262:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760787.834269:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760787.834294:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28696576, count: 4096 00000080:00200000:0.0:1789760787.834300:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28696576, bytes: 4096 00000080:00200000:0.0:1789760787.834307:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760787.834315:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760787.834317:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28696576, 28700672) 0 0 00000080:00200000:0.0:1789760787.834321:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760787.834334:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7006, 7006] 00020000:00200000:0.0:1789760787.834340:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28696576 chunk: [28696576, 28700672) 28700672, 4096 00020000:00200000:0.0:1789760787.834343:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760787.834357:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7006, 7006] 00000008:00010000:0.0:1789760787.834370:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760787.834375:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760787.834384:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28696576, 28700672) append 0 append lockpos 0 00000080:00200000:0.0:1789760787.834394:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28696576=0x1b5e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760787.834436:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760787.834439:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28700672 00000020:00010000:0.0:1789760787.834445:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760787.834449:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.834452:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.834455:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760787.834458:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760787.834463:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760787.834478:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760787.834491:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc000 00000008:00100000:0.0:1789760787.834566:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149500 - object 0x240000400:5 offset 28696576<>28700672 00000100:00100000:0.0:1789760787.834606:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760787.834618:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512760832:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760787.866646:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974324 gen 3 00000100:00100000:0.0:1789760787.866679:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005149500 x1876699512760832/t4294974325(4294974325) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760803 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760787.866717:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:0.0:1789760787.866723:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760787.866732:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760787.866763:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:0.0:1789760787.866785:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512760832:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760787.866929:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7006, 7006] 00000080:00200000:2.0:1789760787.866941:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.866947:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.866958:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28700672 i_blocks 57351 00000020:00200000:2.0:1789760787.866961:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.866969:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.866997:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28700672, count: 4096 00000080:00200000:2.0:1789760787.867003:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28700672, bytes: 4096 00000080:00200000:2.0:1789760787.867011:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.867017:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.867019:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28700672, 28704768) 0 0 00000080:00200000:2.0:1789760787.867023:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.867036:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7007, 7007] 00020000:00200000:2.0:1789760787.867043:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28700672 chunk: [28700672, 28704768) 28704768, 4096 00020000:00200000:2.0:1789760787.867047:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.867063:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7007, 7007] 00000008:00010000:2.0:1789760787.867086:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.867090:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760787.867100:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28700672, 28704768) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.867111:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28700672=0x1b5f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.867157:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.867160:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28704768 00000020:00010000:2.0:1789760787.867166:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.867169:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.867171:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.867174:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.867177:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.867182:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.867187:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.867200:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760787.867276:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 28700672<>28704768 00000100:00100000:3.0:1789760787.867338:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512760960:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760787.900121:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974326 gen 3 00000008:00010000:2.0:1789760787.900173:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:2.0:1789760787.900180:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760787.900187:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760787.900209:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:2.0:1789760787.900234:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512760960:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760787.900479:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7007, 7007] 00000080:00200000:0.0:1789760787.900491:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760787.900496:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760787.900508:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28704768 i_blocks 57351 00000020:00200000:0.0:1789760787.900511:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760787.900519:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760787.900547:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28704768, count: 4096 00000080:00200000:0.0:1789760787.900554:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28704768, bytes: 4096 00000080:00200000:0.0:1789760787.900561:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760787.900568:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760787.900570:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28704768, 28708864) 0 0 00000080:00200000:0.0:1789760787.900574:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760787.900586:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7008, 7008] 00020000:00200000:0.0:1789760787.900593:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28704768 chunk: [28704768, 28708864) 28708864, 4096 00020000:00200000:0.0:1789760787.900597:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760787.900610:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7008, 7008] 00000008:00010000:0.0:1789760787.900624:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760787.900629:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760787.900637:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28704768, 28708864) append 0 append lockpos 0 00000080:00200000:0.0:1789760787.900647:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28704768=0x1b60000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760787.900690:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760787.900693:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28708864 00000020:00010000:0.0:1789760787.900699:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760787.900703:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.900705:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.900709:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760787.900711:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760787.900716:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760787.900722:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760787.900735:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc000 00000008:00100000:0.0:1789760787.900820:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149f80 - object 0x240000400:5 offset 28704768<>28708864 00000100:00100000:1.0:1789760787.901003:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760787.901019:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512761088:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760787.940184:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760787.940195:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.940205:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.940225:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760787.940248:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512761088:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760787.940553:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7008, 7008] 00000080:00200000:0.0:1789760787.940565:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760787.940570:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760787.940580:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28708864 i_blocks 57351 00000020:00200000:0.0:1789760787.940583:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760787.940592:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760787.940617:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28708864, count: 4096 00000080:00200000:0.0:1789760787.940623:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28708864, bytes: 4096 00000080:00200000:0.0:1789760787.940630:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760787.940637:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760787.940639:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28708864, 28712960) 0 0 00000080:00200000:0.0:1789760787.940643:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760787.940655:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7009, 7009] 00020000:00200000:0.0:1789760787.940661:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28708864 chunk: [28708864, 28712960) 28712960, 4096 00020000:00200000:0.0:1789760787.940666:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760787.940680:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7009, 7009] 00000008:00010000:0.0:1789760787.940696:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760787.940700:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760787.940710:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28708864, 28712960) append 0 append lockpos 0 00000080:00200000:0.0:1789760787.940720:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28708864=0x1b61000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760787.940777:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760787.940780:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28712960 00000020:00010000:0.0:1789760787.940834:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760787.940839:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.940842:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760787.940845:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760787.940848:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760787.940854:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760787.940861:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760787.940874:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccf00 00000008:00100000:0.0:1789760787.941164:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514bb80 - object 0x240000400:5 offset 28708864<>28712960 00000100:00100000:1.0:1789760787.941217:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512761216:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760787.975008:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974328 gen 3 00000008:00010000:1.0:1789760787.975058:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760787.975065:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760787.975073:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760787.975104:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760787.975128:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512761216:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760787.975223:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7009, 7009] 00000080:00200000:2.0:1789760787.975234:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760787.975239:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760787.975250:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28712960 i_blocks 57351 00000020:00200000:2.0:1789760787.975254:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760787.975262:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760787.975285:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28712960, count: 4096 00000080:00200000:2.0:1789760787.975291:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28712960, bytes: 4096 00000080:00200000:2.0:1789760787.975299:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760787.975306:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760787.975308:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28712960, 28717056) 0 0 00000080:00200000:2.0:1789760787.975312:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760787.975325:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7010, 7010] 00020000:00200000:2.0:1789760787.975331:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28712960 chunk: [28712960, 28717056) 28717056, 4096 00020000:00200000:2.0:1789760787.975335:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760787.975350:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7010, 7010] 00000008:00010000:2.0:1789760787.975364:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760787.975369:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760787.975377:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28712960, 28717056) append 0 append lockpos 0 00000080:00200000:2.0:1789760787.975387:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28712960=0x1b62000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760787.975430:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760787.975433:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28717056 00000020:00010000:2.0:1789760787.975439:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760787.975443:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.975446:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760787.975449:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760787.975452:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760787.975459:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760787.975464:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760787.975485:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760787.975563:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 28712960<>28717056 00000100:00100000:2.0:1789760787.975604:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760787.975616:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512761344:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760788.005198:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974329 gen 3 00000008:00010000:2.0:1789760788.005232:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:2.0:1789760788.005238:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760788.005247:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760788.005270:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:2.0:1789760788.005293:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512761344:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760788.005392:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7010, 7010] 00000080:00200000:1.0:1789760788.005405:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760788.005411:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760788.005423:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28717056 i_blocks 57351 00000020:00200000:1.0:1789760788.005426:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760788.005435:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760788.005460:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28717056, count: 4096 00000080:00200000:1.0:1789760788.005475:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28717056, bytes: 4096 00000080:00200000:1.0:1789760788.005484:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760788.005491:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760788.005493:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28717056, 28721152) 0 0 00000080:00200000:1.0:1789760788.005497:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760788.005509:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7011, 7011] 00020000:00200000:1.0:1789760788.005516:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28717056 chunk: [28717056, 28721152) 28721152, 4096 00020000:00200000:1.0:1789760788.005520:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760788.005536:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7011, 7011] 00000008:00010000:1.0:1789760788.005550:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760788.005555:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760788.005564:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28717056, 28721152) append 0 append lockpos 0 00000080:00200000:1.0:1789760788.005575:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28717056=0x1b63000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760788.005621:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760788.005623:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28721152 00000020:00010000:1.0:1789760788.005630:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760788.005634:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.005637:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.005640:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760788.005642:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760788.005649:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760788.005654:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760788.005668:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760788.005738:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94a80 - object 0x240000400:5 offset 28717056<>28721152 00000100:00100000:0.0:1789760788.005824:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512761472:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760788.032505:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974330 gen 3 00000008:00010000:0.0:1789760788.032539:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760788.032545:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760788.032553:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760788.032575:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760788.032597:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512761472:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760788.032699:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7011, 7011] 00000080:00200000:2.0:1789760788.032711:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.032716:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.032727:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28721152 i_blocks 57351 00000020:00200000:2.0:1789760788.032731:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.032738:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.032772:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28721152, count: 4096 00000080:00200000:2.0:1789760788.032779:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28721152, bytes: 4096 00000080:00200000:2.0:1789760788.032796:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.032804:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.032806:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28721152, 28725248) 0 0 00000080:00200000:2.0:1789760788.032810:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.032822:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7012, 7012] 00020000:00200000:2.0:1789760788.032829:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28721152 chunk: [28721152, 28725248) 28725248, 4096 00020000:00200000:2.0:1789760788.032833:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.032847:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7012, 7012] 00000008:00010000:2.0:1789760788.032861:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.032865:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760788.032874:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28721152, 28725248) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.032885:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28721152=0x1b64000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.032970:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.032973:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28725248 00000020:00010000:2.0:1789760788.032979:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.032982:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.032984:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.032987:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.032990:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.032996:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.033001:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.033011:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760788.033090:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 28721152<>28725248 00000100:00100000:3.0:1789760788.033149:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512761600:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760788.063938:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974331 gen 3 00000008:00010000:2.0:1789760788.063975:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:2.0:1789760788.063981:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760788.063989:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760788.064012:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:2.0:1789760788.064035:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512761600:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760788.064214:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7012, 7012] 00000080:00200000:0.0:1789760788.064227:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.064232:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.064243:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28725248 i_blocks 57351 00000020:00200000:0.0:1789760788.064246:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.064254:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.064280:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28725248, count: 4096 00000080:00200000:0.0:1789760788.064285:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28725248, bytes: 4096 00000080:00200000:0.0:1789760788.064294:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.064301:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760788.064303:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28725248, 28729344) 0 0 00000080:00200000:0.0:1789760788.064307:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.064320:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7013, 7013] 00020000:00200000:0.0:1789760788.064327:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28725248 chunk: [28725248, 28729344) 28729344, 4096 00020000:00200000:0.0:1789760788.064331:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.064345:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7013, 7013] 00000008:00010000:0.0:1789760788.064358:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.064363:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760788.064371:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28725248, 28729344) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.064382:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28725248=0x1b65000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.064424:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.064427:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28729344 00000020:00010000:0.0:1789760788.064433:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.064436:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.064439:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.064442:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.064444:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.064450:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.064456:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.064525:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cca80 00000008:00100000:0.0:1789760788.064599:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149c00 - object 0x240000400:5 offset 28725248<>28729344 00000100:00100000:1.0:1789760788.064655:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512761728:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760788.095270:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760788.095280:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.095290:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.095312:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760788.095335:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512761728:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760788.095428:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7013, 7013] 00000080:00200000:0.0:1789760788.095440:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760788.095444:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760788.095455:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28729344 i_blocks 57351 00000020:00200000:0.0:1789760788.095458:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760788.095474:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760788.095499:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28729344, count: 4096 00000080:00200000:0.0:1789760788.095505:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28729344, bytes: 4096 00000080:00200000:0.0:1789760788.095512:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760788.095520:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760788.095522:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28729344, 28733440) 0 0 00000080:00200000:0.0:1789760788.095526:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760788.095538:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7014, 7014] 00020000:00200000:0.0:1789760788.095545:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28729344 chunk: [28729344, 28733440) 28733440, 4096 00020000:00200000:0.0:1789760788.095549:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760788.095563:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7014, 7014] 00000008:00010000:0.0:1789760788.095576:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760788.095580:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760788.095588:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28729344, 28733440) append 0 append lockpos 0 00000080:00200000:0.0:1789760788.095598:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28729344=0x1b66000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760788.095638:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760788.095641:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28733440 00000020:00010000:0.0:1789760788.095648:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760788.095651:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.095653:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760788.095657:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760788.095659:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760788.095665:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760788.095671:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760788.095684:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc540 00000008:00100000:0.0:1789760788.096013:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148000 - object 0x240000400:5 offset 28729344<>28733440 00000100:00100000:1.0:1789760788.096059:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760788.096226:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512761856:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760788.127115:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974333 gen 3 00000008:00010000:1.0:1789760788.127164:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760788.127170:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.127179:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.127208:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760788.127233:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512761856:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760788.127499:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7014, 7014] 00000080:00200000:2.0:1789760788.127513:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.127518:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.127531:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28733440 i_blocks 57351 00000020:00200000:2.0:1789760788.127535:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.127545:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.127696:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28733440, count: 4096 00000080:00200000:2.0:1789760788.127704:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28733440, bytes: 4096 00000080:00200000:2.0:1789760788.127713:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.127720:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.127722:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28733440, 28737536) 0 0 00000080:00200000:2.0:1789760788.127726:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.127739:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7015, 7015] 00020000:00200000:2.0:1789760788.127746:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28733440 chunk: [28733440, 28737536) 28737536, 4096 00020000:00200000:2.0:1789760788.127749:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.127765:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7015, 7015] 00000008:00010000:2.0:1789760788.127778:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.127783:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760788.127792:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28733440, 28737536) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.127803:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28733440=0x1b67000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.127846:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.127849:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28737536 00000020:00010000:2.0:1789760788.127855:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.127858:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.127861:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.127864:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.127867:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.127873:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.127879:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.127894:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760788.128012:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 28733440<>28737536 00000100:00100000:3.0:1789760788.128216:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512761984:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760788.166514:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974334 gen 3 00000008:00010000:3.0:1789760788.166550:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760788.166955:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.166965:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.166984:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760788.167010:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512761984:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760788.167165:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7015, 7015] 00000080:00200000:2.0:1789760788.167179:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.167184:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.167197:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28737536 i_blocks 57351 00000020:00200000:2.0:1789760788.167267:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.167276:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.167300:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28737536, count: 4096 00000080:00200000:2.0:1789760788.167306:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28737536, bytes: 4096 00000080:00200000:2.0:1789760788.167314:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.167321:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.167323:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28737536, 28741632) 0 0 00000080:00200000:2.0:1789760788.167327:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.167339:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7016, 7016] 00020000:00200000:2.0:1789760788.167345:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28737536 chunk: [28737536, 28741632) 28741632, 4096 00020000:00200000:2.0:1789760788.167349:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.167363:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7016, 7016] 00000008:00010000:2.0:1789760788.167375:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.167380:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760788.167388:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28737536, 28741632) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.167398:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28737536=0x1b68000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.167439:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.167441:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28741632 00000020:00010000:2.0:1789760788.167448:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.167451:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.167454:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.167457:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.167460:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.167480:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.167486:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.167500:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760788.167583:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 28737536<>28741632 00000100:00100000:3.0:1789760788.167954:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512762112:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760788.206211:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974335 gen 3 00000008:00010000:3.0:1789760788.206243:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760788.206249:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.206257:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.206310:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760788.206329:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512762112:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760788.206756:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7016, 7016] 00000080:00200000:2.0:1789760788.206768:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.206773:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.206784:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28741632 i_blocks 57351 00000020:00200000:2.0:1789760788.206787:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.206795:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.206818:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28741632, count: 4096 00000080:00200000:2.0:1789760788.206824:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28741632, bytes: 4096 00000080:00200000:2.0:1789760788.206831:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.206838:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.206840:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28741632, 28745728) 0 0 00000080:00200000:2.0:1789760788.206844:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.206855:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7017, 7017] 00020000:00200000:2.0:1789760788.206861:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28741632 chunk: [28741632, 28745728) 28745728, 4096 00020000:00200000:2.0:1789760788.206865:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.206877:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7017, 7017] 00000008:00010000:2.0:1789760788.206889:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.206894:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760788.206922:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28741632, 28745728) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.206932:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28741632=0x1b69000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.206971:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.206973:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28745728 00000020:00010000:2.0:1789760788.206979:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.206982:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.206985:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.206988:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.206991:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.206996:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.207001:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.207014:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760788.207151:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 28741632<>28745728 00000100:00100000:3.0:1789760788.207204:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512762240:192.168.203.156@tcp:4:dd.0 00010000:00010000:1.0:1789760788.253773:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760788.262560:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974336 gen 3 00000008:00010000:3.0:1789760788.262600:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760788.262606:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.262615:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.262685:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760788.262712:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512762240:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760788.262967:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7017, 7017] 00000080:00200000:2.0:1789760788.262980:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.262986:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.262998:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28745728 i_blocks 57351 00000020:00200000:2.0:1789760788.263001:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.263010:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.263036:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28745728, count: 4096 00000080:00200000:2.0:1789760788.263042:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28745728, bytes: 4096 00000080:00200000:2.0:1789760788.263050:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.263057:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.263059:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28745728, 28749824) 0 0 00000080:00200000:2.0:1789760788.263064:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.263086:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7018, 7018] 00020000:00200000:2.0:1789760788.263093:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28745728 chunk: [28745728, 28749824) 28749824, 4096 00020000:00200000:2.0:1789760788.263097:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.263112:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7018, 7018] 00000008:00010000:2.0:1789760788.263126:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.263130:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760788.263140:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28745728, 28749824) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.263150:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28745728=0x1b6a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.263273:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.263276:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28749824 00000020:00010000:2.0:1789760788.263283:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.263287:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.263290:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.263293:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.263296:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.263302:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.263310:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.263327:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760788.263406:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 28745728<>28749824 00000100:00100000:3.0:1789760788.263477:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512762368:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760788.302878:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760788.302891:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.302993:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.303054:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760788.303087:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512762368:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760788.303248:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7018, 7018] 00000080:00200000:2.0:1789760788.303261:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.303266:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.303278:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28749824 i_blocks 57351 00000020:00200000:2.0:1789760788.303281:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.303289:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.303314:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28749824, count: 4096 00000080:00200000:2.0:1789760788.303320:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28749824, bytes: 4096 00000080:00200000:2.0:1789760788.303327:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.303333:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.303335:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28749824, 28753920) 0 0 00000080:00200000:2.0:1789760788.303339:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.303354:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7019, 7019] 00020000:00200000:2.0:1789760788.303361:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28749824 chunk: [28749824, 28753920) 28753920, 4096 00020000:00200000:2.0:1789760788.303364:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.303380:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7019, 7019] 00000008:00010000:2.0:1789760788.303394:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.303398:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760788.303407:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28749824, 28753920) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.303416:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28749824=0x1b6b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.303463:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.303539:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28753920 00000020:00010000:2.0:1789760788.303547:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.303551:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.303554:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.303556:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.303559:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.303566:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.303571:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.303586:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760788.303660:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 28749824<>28753920 00000100:00100000:3.0:1789760788.303722:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512762496:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760788.334947:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974338 gen 3 00000008:00010000:3.0:1789760788.334992:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760788.334999:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.335006:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.335023:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760788.335046:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512762496:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760788.335172:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7019, 7019] 00000080:00200000:2.0:1789760788.335184:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.335189:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.335199:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28753920 i_blocks 57351 00000020:00200000:2.0:1789760788.335202:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.335209:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.335234:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28753920, count: 4096 00000080:00200000:2.0:1789760788.335240:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28753920, bytes: 4096 00000080:00200000:2.0:1789760788.335247:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.335254:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.335256:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28753920, 28758016) 0 0 00000080:00200000:2.0:1789760788.335260:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.335275:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7020, 7020] 00020000:00200000:2.0:1789760788.335281:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28753920 chunk: [28753920, 28758016) 28758016, 4096 00020000:00200000:2.0:1789760788.335285:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.335300:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7020, 7020] 00000008:00010000:2.0:1789760788.335313:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.335318:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760788.335328:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28753920, 28758016) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.335339:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28753920=0x1b6c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.335379:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.335382:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28758016 00000020:00010000:2.0:1789760788.335388:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.335391:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.335394:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.335396:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.335399:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.335405:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.335410:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.335423:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760788.335509:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 28753920<>28758016 00000100:00100000:3.0:1789760788.335881:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512762624:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760788.405815:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974339 gen 3 00000008:00010000:3.0:1789760788.405851:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760788.405857:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.405865:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.405885:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760788.405937:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512762624:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760788.406058:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7020, 7020] 00000080:00200000:2.0:1789760788.406085:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.406091:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.406103:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28758016 i_blocks 57351 00000020:00200000:2.0:1789760788.406106:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.406115:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.406140:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28758016, count: 4096 00000080:00200000:2.0:1789760788.406147:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28758016, bytes: 4096 00000080:00200000:2.0:1789760788.406154:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.406161:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.406163:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28758016, 28762112) 0 0 00000080:00200000:2.0:1789760788.406167:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.406180:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7021, 7021] 00020000:00200000:2.0:1789760788.406186:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28758016 chunk: [28758016, 28762112) 28762112, 4096 00020000:00200000:2.0:1789760788.406190:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.406205:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7021, 7021] 00000008:00010000:2.0:1789760788.406219:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.406224:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760788.406233:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28758016, 28762112) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.406244:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28758016=0x1b6d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.406288:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.406290:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28762112 00000020:00010000:2.0:1789760788.406297:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.406299:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.406302:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.406305:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.406307:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.406314:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.406321:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.406334:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760788.406424:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 28758016<>28762112 00000100:00100000:3.0:1789760788.406585:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512762752:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760788.459291:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760788.459306:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.459315:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.459333:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760788.459355:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512762752:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760788.459436:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7021, 7021] 00000080:00200000:2.0:1789760788.459448:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.459453:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.459464:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28762112 i_blocks 57351 00000020:00200000:2.0:1789760788.459476:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.459484:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.459510:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28762112, count: 4096 00000080:00200000:2.0:1789760788.459517:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28762112, bytes: 4096 00000080:00200000:2.0:1789760788.459525:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.459531:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.459533:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28762112, 28766208) 0 0 00000080:00200000:2.0:1789760788.459537:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.459551:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7022, 7022] 00020000:00200000:2.0:1789760788.459559:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28762112 chunk: [28762112, 28766208) 28766208, 4096 00020000:00200000:2.0:1789760788.459563:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.459579:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7022, 7022] 00000008:00010000:2.0:1789760788.459596:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.459603:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760788.459612:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28762112, 28766208) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.459622:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28762112=0x1b6e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.459663:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.459666:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28766208 00000020:00010000:2.0:1789760788.459673:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.459676:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.459678:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.459681:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.459683:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.459689:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.459694:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.459710:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760788.459946:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 28762112<>28766208 00000100:00100000:3.0:1789760788.460090:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512762880:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760788.508944:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974341 gen 3 00000008:00010000:3.0:1789760788.508992:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760788.508998:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.509008:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.509270:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760788.509296:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512762880:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760788.509424:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7022, 7022] 00000080:00200000:2.0:1789760788.509436:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760788.509442:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760788.509454:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28766208 i_blocks 57351 00000020:00200000:2.0:1789760788.509457:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760788.509533:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760788.509560:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28766208, count: 4096 00000080:00200000:2.0:1789760788.509566:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28766208, bytes: 4096 00000080:00200000:2.0:1789760788.509575:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760788.509582:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760788.509584:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28766208, 28770304) 0 0 00000080:00200000:2.0:1789760788.509588:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760788.509601:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7023, 7023] 00020000:00200000:2.0:1789760788.509607:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28766208 chunk: [28766208, 28770304) 28770304, 4096 00020000:00200000:2.0:1789760788.509610:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760788.509625:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7023, 7023] 00000008:00010000:2.0:1789760788.509637:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760788.509641:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760788.509650:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28766208, 28770304) append 0 append lockpos 0 00000080:00200000:2.0:1789760788.509661:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28766208=0x1b6f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760788.509703:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760788.509706:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28770304 00000020:00010000:2.0:1789760788.509712:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760788.509715:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.509718:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760788.509721:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760788.509724:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760788.509730:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760788.509735:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760788.509861:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760788.510035:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 28766208<>28770304 00000100:00100000:2.0:1789760788.510198:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512763008:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760788.568542:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974342 gen 3 00000008:00010000:3.0:1789760788.568583:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760788.568589:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.568598:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.568677:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760788.568702:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512763008:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760788.568840:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7023, 7023] 00000080:00200000:1.0:1789760788.568852:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760788.568857:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760788.568868:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28770304 i_blocks 57351 00000020:00200000:1.0:1789760788.568872:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760788.568879:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760788.568929:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28770304, count: 4096 00000080:00200000:1.0:1789760788.568936:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28770304, bytes: 4096 00000080:00200000:1.0:1789760788.568945:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760788.568952:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760788.568954:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28770304, 28774400) 0 0 00000080:00200000:1.0:1789760788.568958:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760788.568970:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7024, 7024] 00020000:00200000:1.0:1789760788.568977:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28770304 chunk: [28770304, 28774400) 28774400, 4096 00020000:00200000:1.0:1789760788.568981:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760788.568996:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7024, 7024] 00000008:00010000:1.0:1789760788.569011:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760788.569016:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760788.569028:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28770304, 28774400) append 0 append lockpos 0 00000080:00200000:1.0:1789760788.569038:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28770304=0x1b70000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760788.569092:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760788.569094:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28774400 00000020:00010000:1.0:1789760788.569101:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760788.569104:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.569106:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.569110:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760788.569112:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760788.569118:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760788.569124:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760788.569138:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760788.569216:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97800 - object 0x240000400:5 offset 28770304<>28774400 00000100:00100000:0.0:1789760788.569422:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512763136:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760788.640707:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974343 gen 3 00000008:00010000:0.0:1789760788.640741:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760788.640747:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760788.640756:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760788.640776:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760788.640798:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512763136:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760788.641018:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7024, 7024] 00000080:00200000:1.0:1789760788.641033:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760788.641038:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760788.641051:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28774400 i_blocks 57351 00000020:00200000:1.0:1789760788.641054:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760788.641063:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760788.641103:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28774400, count: 4096 00000080:00200000:1.0:1789760788.641109:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28774400, bytes: 4096 00000080:00200000:1.0:1789760788.641117:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760788.641125:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760788.641127:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28774400, 28778496) 0 0 00000080:00200000:1.0:1789760788.641131:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760788.641144:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7025, 7025] 00020000:00200000:1.0:1789760788.641150:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28774400 chunk: [28774400, 28778496) 28778496, 4096 00020000:00200000:1.0:1789760788.641154:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760788.641168:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7025, 7025] 00000008:00010000:1.0:1789760788.641182:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760788.641187:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760788.641196:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28774400, 28778496) append 0 append lockpos 0 00000080:00200000:1.0:1789760788.641207:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28774400=0x1b71000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760788.641265:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760788.641268:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28778496 00000020:00010000:1.0:1789760788.641285:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760788.641288:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.641290:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.641294:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760788.641296:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760788.641302:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760788.641309:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760788.641323:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760788.641398:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95c00 - object 0x240000400:5 offset 28774400<>28778496 00000100:00100000:0.0:1789760788.641671:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760788.641686:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512763264:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760788.706068:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974344 gen 3 00000008:00010000:0.0:1789760788.706121:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760788.706128:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760788.706138:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760788.706158:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760788.706191:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512763264:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760788.706709:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7025, 7025] 00000080:00200000:1.0:1789760788.706726:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760788.706731:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760788.706743:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28778496 i_blocks 57351 00000020:00200000:1.0:1789760788.706747:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760788.706755:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760788.706782:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28778496, count: 4096 00000080:00200000:1.0:1789760788.706788:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28778496, bytes: 4096 00000080:00200000:1.0:1789760788.706797:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760788.706805:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760788.706807:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28778496, 28782592) 0 0 00000080:00200000:1.0:1789760788.706811:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760788.706825:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7026, 7026] 00020000:00200000:1.0:1789760788.706832:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28778496 chunk: [28778496, 28782592) 28782592, 4096 00020000:00200000:1.0:1789760788.706836:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760788.706856:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7026, 7026] 00000008:00010000:1.0:1789760788.706870:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760788.706875:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760788.706883:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28778496, 28782592) append 0 append lockpos 0 00000080:00200000:1.0:1789760788.706894:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28778496=0x1b72000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760788.706972:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760788.706974:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28782592 00000020:00010000:1.0:1789760788.706981:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760788.706984:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.706987:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.706990:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760788.706992:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760788.706999:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760788.707004:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760788.707019:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760788.707105:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 28778496<>28782592 00000100:00100000:1.0:1789760788.707150:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760788.707231:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512763392:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760788.799941:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974345 gen 3 00000008:00010000:1.0:1789760788.799981:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760788.799987:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.799995:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.800019:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760788.800042:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512763392:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760788.800400:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7026, 7026] 00000080:00200000:3.0:1789760788.800414:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760788.800419:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760788.800431:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28782592 i_blocks 57351 00000020:00200000:3.0:1789760788.800435:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760788.800442:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760788.800469:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28782592, count: 4096 00000080:00200000:3.0:1789760788.800476:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28782592, bytes: 4096 00000080:00200000:3.0:1789760788.800485:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760788.800493:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760788.800494:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28782592, 28786688) 0 0 00000080:00200000:3.0:1789760788.800498:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760788.800510:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7027, 7027] 00020000:00200000:3.0:1789760788.800517:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28782592 chunk: [28782592, 28786688) 28786688, 4096 00020000:00200000:3.0:1789760788.800520:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760788.800535:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7027, 7027] 00000008:00010000:3.0:1789760788.800550:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760788.800555:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760788.800564:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28782592, 28786688) append 0 append lockpos 0 00000080:00200000:3.0:1789760788.800575:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28782592=0x1b73000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760788.800617:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760788.800621:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28786688 00000020:00010000:3.0:1789760788.800627:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760788.800631:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760788.800633:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760788.800636:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760788.800639:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760788.800645:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760788.800651:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760788.800665:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760788.800766:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af800 - object 0x240000400:5 offset 28782592<>28786688 00000100:00100000:2.0:1789760788.800837:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512763520:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760788.852861:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974346 gen 3 00000008:00010000:3.0:1789760788.852897:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760788.852930:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760788.852939:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760788.852962:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760788.852985:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512763520:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760788.853101:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7027, 7027] 00000080:00200000:1.0:1789760788.853113:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760788.853119:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760788.853130:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28786688 i_blocks 57351 00000020:00200000:1.0:1789760788.853133:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760788.853141:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760788.853166:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28786688, count: 4096 00000080:00200000:1.0:1789760788.853171:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28786688, bytes: 4096 00000080:00200000:1.0:1789760788.853179:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760788.853185:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760788.853187:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28786688, 28790784) 0 0 00000080:00200000:1.0:1789760788.853190:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760788.853203:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7028, 7028] 00020000:00200000:1.0:1789760788.853209:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28786688 chunk: [28786688, 28790784) 28790784, 4096 00020000:00200000:1.0:1789760788.853213:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760788.853227:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7028, 7028] 00000008:00010000:1.0:1789760788.853241:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760788.853246:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760788.853254:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28786688, 28790784) append 0 append lockpos 0 00000080:00200000:1.0:1789760788.853263:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28786688=0x1b74000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760788.853307:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760788.853309:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28790784 00000020:00010000:1.0:1789760788.853315:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760788.853319:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.853322:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.853324:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760788.853327:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760788.853333:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760788.853338:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760788.853352:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760788.853428:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 28786688<>28790784 00000100:00100000:0.0:1789760788.853652:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760788.853667:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512763648:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760788.889737:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760788.889748:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760788.889757:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760788.889777:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760788.889802:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512763648:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760788.889933:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7028, 7028] 00000080:00200000:1.0:1789760788.889947:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760788.889952:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760788.889963:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28790784 i_blocks 57351 00000020:00200000:1.0:1789760788.890032:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760788.890040:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760788.890067:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28790784, count: 4096 00000080:00200000:1.0:1789760788.890083:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28790784, bytes: 4096 00000080:00200000:1.0:1789760788.890153:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760788.890161:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760788.890163:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28790784, 28794880) 0 0 00000080:00200000:1.0:1789760788.890167:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760788.890181:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7029, 7029] 00020000:00200000:1.0:1789760788.890187:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28790784 chunk: [28790784, 28794880) 28794880, 4096 00020000:00200000:1.0:1789760788.890190:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760788.890205:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7029, 7029] 00000008:00010000:1.0:1789760788.890218:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760788.890223:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760788.890232:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28790784, 28794880) append 0 append lockpos 0 00000080:00200000:1.0:1789760788.890242:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28790784=0x1b75000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760788.890282:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760788.890285:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28794880 00000020:00010000:1.0:1789760788.890291:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760788.890294:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.890297:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760788.890300:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760788.890303:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760788.890309:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760788.890315:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760788.890328:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760788.890401:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 28790784<>28794880 00000100:00100000:0.0:1789760788.890467:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512763776:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760788.944217:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974348 gen 3 00000008:00010000:1.0:1789760788.944264:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760788.944271:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760788.944278:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760788.944301:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760788.944322:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512763776:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760788.944475:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7029, 7029] 00000080:00200000:3.0:1789760788.944490:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760788.944495:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760788.944507:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28794880 i_blocks 57351 00000020:00200000:3.0:1789760788.944510:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760788.944518:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760788.944547:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28794880, count: 4096 00000080:00200000:3.0:1789760788.944553:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28794880, bytes: 4096 00000080:00200000:3.0:1789760788.944562:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760788.944569:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760788.944571:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28794880, 28798976) 0 0 00000080:00200000:3.0:1789760788.944575:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760788.944587:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7030, 7030] 00020000:00200000:3.0:1789760788.944594:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28794880 chunk: [28794880, 28798976) 28798976, 4096 00020000:00200000:3.0:1789760788.944598:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760788.944611:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7030, 7030] 00000008:00010000:3.0:1789760788.944624:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760788.944628:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760788.944636:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28794880, 28798976) append 0 append lockpos 0 00000080:00200000:3.0:1789760788.944646:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28794880=0x1b76000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760788.944686:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760788.944689:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28798976 00000020:00010000:3.0:1789760788.944696:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760788.944700:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760788.944702:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760788.944706:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760788.944708:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760788.944714:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760788.944720:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760788.944733:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760788.944807:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af800 - object 0x240000400:5 offset 28794880<>28798976 00000100:00100000:3.0:1789760788.944845:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760788.944859:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512763904:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760789.000341:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974349 gen 3 00000008:00010000:3.0:1789760789.000377:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760789.000383:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.000392:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.000452:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760789.000477:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512763904:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760789.000932:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7030, 7030] 00000080:00200000:1.0:1789760789.000945:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.000949:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.000961:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28798976 i_blocks 57351 00000020:00200000:1.0:1789760789.000964:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.000972:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.000996:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28798976, count: 4096 00000080:00200000:1.0:1789760789.001002:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28798976, bytes: 4096 00000080:00200000:1.0:1789760789.001009:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.001016:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760789.001018:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28798976, 28803072) 0 0 00000080:00200000:1.0:1789760789.001021:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.001033:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7031, 7031] 00020000:00200000:1.0:1789760789.001039:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28798976 chunk: [28798976, 28803072) 28803072, 4096 00020000:00200000:1.0:1789760789.001043:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.001057:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7031, 7031] 00000008:00010000:1.0:1789760789.001082:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.001087:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760789.001095:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28798976, 28803072) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.001107:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28798976=0x1b77000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.001150:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.001152:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28803072 00000020:00010000:1.0:1789760789.001159:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.001162:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.001165:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.001169:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.001171:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.001177:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.001183:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.001196:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760789.001271:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 28798976<>28803072 00000100:00100000:1.0:1789760789.001310:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760789.001326:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512764032:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760789.038955:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974350 gen 3 00000008:00010000:1.0:1789760789.038993:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:1.0:1789760789.038999:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.039006:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.039087:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:1.0:1789760789.039112:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512764032:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760789.039182:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7031, 7031] 00000080:00200000:3.0:1789760789.039194:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760789.039199:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760789.039211:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28803072 i_blocks 57351 00000020:00200000:3.0:1789760789.039215:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760789.039223:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760789.039405:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28803072, count: 4096 00000080:00200000:3.0:1789760789.039412:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28803072, bytes: 4096 00000080:00200000:3.0:1789760789.039421:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760789.039428:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760789.039430:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28803072, 28807168) 0 0 00000080:00200000:3.0:1789760789.039434:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760789.039447:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7032, 7032] 00020000:00200000:3.0:1789760789.039454:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28803072 chunk: [28803072, 28807168) 28807168, 4096 00020000:00200000:3.0:1789760789.039457:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760789.039541:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7032, 7032] 00000008:00010000:3.0:1789760789.039555:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760789.039559:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760789.039568:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28803072, 28807168) append 0 append lockpos 0 00000080:00200000:3.0:1789760789.039578:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28803072=0x1b78000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760789.039620:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760789.039623:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28807168 00000020:00010000:3.0:1789760789.039629:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760789.039633:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.039635:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.039638:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760789.039641:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760789.039647:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760789.039653:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760789.039668:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760789.039855:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af800 - object 0x240000400:5 offset 28803072<>28807168 00000100:00100000:2.0:1789760789.039964:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512764160:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760789.073299:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760789.073311:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.073320:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.073342:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760789.073366:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512764160:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760789.073453:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7032, 7032] 00000080:00200000:1.0:1789760789.073467:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.073482:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.073495:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28807168 i_blocks 57351 00000020:00200000:1.0:1789760789.073498:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.073507:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.073535:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28807168, count: 4096 00000080:00200000:1.0:1789760789.073541:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28807168, bytes: 4096 00000080:00200000:1.0:1789760789.073549:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.073558:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760789.073560:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28807168, 28811264) 0 0 00000080:00200000:1.0:1789760789.073564:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.073578:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7033, 7033] 00020000:00200000:1.0:1789760789.073585:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28807168 chunk: [28807168, 28811264) 28811264, 4096 00020000:00200000:1.0:1789760789.073588:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.073605:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7033, 7033] 00000008:00010000:1.0:1789760789.073620:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.073627:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760789.073635:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28807168, 28811264) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.073645:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28807168=0x1b79000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.073693:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.073695:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28811264 00000020:00010000:1.0:1789760789.073701:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.073704:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.073706:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.073709:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.073712:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.073718:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.073723:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.073750:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760789.074017:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95f80 - object 0x240000400:5 offset 28807168<>28811264 00000100:00100000:0.0:1789760789.074120:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512764288:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760789.101161:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974352 gen 3 00000008:00010000:0.0:1789760789.101210:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760789.101216:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.101224:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.101245:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760789.101268:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512764288:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760789.101649:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7033, 7033] 00000080:00200000:1.0:1789760789.101660:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.101665:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.101677:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28811264 i_blocks 57351 00000020:00200000:1.0:1789760789.101681:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.101690:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.101715:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28811264, count: 4096 00000080:00200000:1.0:1789760789.101722:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28811264, bytes: 4096 00000080:00200000:1.0:1789760789.101729:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.101747:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760789.101749:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28811264, 28815360) 0 0 00000080:00200000:1.0:1789760789.101753:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.101766:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7034, 7034] 00020000:00200000:1.0:1789760789.101774:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28811264 chunk: [28811264, 28815360) 28815360, 4096 00020000:00200000:1.0:1789760789.101778:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.101794:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7034, 7034] 00000008:00010000:1.0:1789760789.101807:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.101812:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760789.101821:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28811264, 28815360) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.101832:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28811264=0x1b7a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.101879:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.101883:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28815360 00000020:00010000:1.0:1789760789.101890:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.101893:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.101896:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.101937:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.101941:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.101948:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.101954:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.101969:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760789.102046:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96300 - object 0x240000400:5 offset 28811264<>28815360 00000100:00100000:0.0:1789760789.102139:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760789.102154:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512764416:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760789.134430:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760789.134441:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.134450:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.134477:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760789.134499:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512764416:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760789.134732:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7034, 7034] 00000080:00200000:1.0:1789760789.134753:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.134758:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.134769:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28815360 i_blocks 57351 00000020:00200000:1.0:1789760789.134772:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.134780:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.134810:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28815360, count: 4096 00000080:00200000:1.0:1789760789.134816:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28815360, bytes: 4096 00000080:00200000:1.0:1789760789.134823:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.134830:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760789.134832:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28815360, 28819456) 0 0 00000080:00200000:1.0:1789760789.134836:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.134849:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7035, 7035] 00020000:00200000:1.0:1789760789.134855:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28815360 chunk: [28815360, 28819456) 28819456, 4096 00020000:00200000:1.0:1789760789.134859:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.134872:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7035, 7035] 00000008:00010000:1.0:1789760789.134885:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.134889:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760789.134923:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28815360, 28819456) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.134934:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28815360=0x1b7b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.134978:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.134980:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28819456 00000020:00010000:1.0:1789760789.134986:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.134989:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.134992:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.134995:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.134998:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.135004:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.135008:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.135021:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760789.135207:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94700 - object 0x240000400:5 offset 28815360<>28819456 00000100:00100000:0.0:1789760789.135292:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512764544:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760789.173213:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974353 gen 3 00000100:00100000:0.0:1789760789.173243:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc005c94700 x1876699512764544/t4294974354(4294974354) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760805 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760789.173281:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760789.173287:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.173295:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.173318:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760789.173338:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512764544:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760789.173426:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7035, 7035] 00000080:00200000:1.0:1789760789.173439:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.173445:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.173457:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28819456 i_blocks 57351 00000020:00200000:1.0:1789760789.173471:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.173480:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.173506:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28819456, count: 4096 00000080:00200000:1.0:1789760789.173513:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28819456, bytes: 4096 00000080:00200000:1.0:1789760789.173521:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.173529:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760789.173531:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28819456, 28823552) 0 0 00000080:00200000:1.0:1789760789.173535:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.173550:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7036, 7036] 00020000:00200000:1.0:1789760789.173557:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28819456 chunk: [28819456, 28823552) 28823552, 4096 00020000:00200000:1.0:1789760789.173561:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.173578:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7036, 7036] 00000008:00010000:1.0:1789760789.173591:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.173596:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760789.173605:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28819456, 28823552) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.173616:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28819456=0x1b7c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.173659:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.173662:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28823552 00000020:00010000:1.0:1789760789.173668:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.173672:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.173675:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.173678:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.173681:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.173688:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.173694:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.173708:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760789.173795:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95500 - object 0x240000400:5 offset 28819456<>28823552 00000100:00100000:0.0:1789760789.174090:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760789.174104:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512764672:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760789.212729:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974355 gen 3 00000008:00010000:0.0:1789760789.212773:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760789.212780:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.212799:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.212818:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760789.212841:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512764672:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760789.213151:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7036, 7036] 00000080:00200000:1.0:1789760789.213164:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.213169:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.213180:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28823552 i_blocks 57351 00000020:00200000:1.0:1789760789.213183:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.213191:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.213216:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28823552, count: 4096 00000080:00200000:1.0:1789760789.213221:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28823552, bytes: 4096 00000080:00200000:1.0:1789760789.213228:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.213235:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760789.213237:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28823552, 28827648) 0 0 00000080:00200000:1.0:1789760789.213240:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.213253:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7037, 7037] 00020000:00200000:1.0:1789760789.213259:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28823552 chunk: [28823552, 28827648) 28827648, 4096 00020000:00200000:1.0:1789760789.213264:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.213279:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7037, 7037] 00000008:00010000:1.0:1789760789.213292:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.213297:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760789.213306:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28823552, 28827648) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.213317:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28823552=0x1b7d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.213361:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.213364:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28827648 00000020:00010000:1.0:1789760789.213371:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.213374:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.213377:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.213380:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.213382:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.213388:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.213394:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.213408:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760789.213496:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94380 - object 0x240000400:5 offset 28823552<>28827648 00000100:00100000:0.0:1789760789.213645:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512764800:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760789.251707:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974356 gen 3 00000008:00010000:0.0:1789760789.251744:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760789.251750:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.251759:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.251780:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760789.251813:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512764800:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760789.252019:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7037, 7037] 00000080:00200000:1.0:1789760789.252032:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.252038:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.252049:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28827648 i_blocks 57351 00000020:00200000:1.0:1789760789.252053:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.252062:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.252098:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28827648, count: 4096 00000080:00200000:1.0:1789760789.252104:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28827648, bytes: 4096 00000080:00200000:1.0:1789760789.252111:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.252118:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760789.252120:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28827648, 28831744) 0 0 00000080:00200000:1.0:1789760789.252123:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.252135:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7038, 7038] 00020000:00200000:1.0:1789760789.252141:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28827648 chunk: [28827648, 28831744) 28831744, 4096 00020000:00200000:1.0:1789760789.252146:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.252159:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7038, 7038] 00000008:00010000:1.0:1789760789.252172:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.252177:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760789.252185:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28827648, 28831744) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.252195:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28827648=0x1b7e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.252237:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.252240:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28831744 00000020:00010000:1.0:1789760789.252247:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.252249:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.252252:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.252255:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.252258:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.252263:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.252269:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.252282:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760789.252360:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96a00 - object 0x240000400:5 offset 28827648<>28831744 00000100:00100000:0.0:1789760789.252451:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760789.252474:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512764928:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760789.277236:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760789.289966:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974357 gen 3 00000008:00010000:0.0:1789760789.290001:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760789.290007:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.290015:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.290035:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760789.290059:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512764928:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760789.290283:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7038, 7038] 00000080:00200000:1.0:1789760789.290295:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.290300:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.290311:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28831744 i_blocks 57351 00000020:00200000:1.0:1789760789.290314:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.290322:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.290346:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28831744, count: 4096 00000080:00200000:1.0:1789760789.290352:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28831744, bytes: 4096 00000080:00200000:1.0:1789760789.290360:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.290367:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760789.290369:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28831744, 28835840) 0 0 00000080:00200000:1.0:1789760789.290373:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.290385:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7039, 7039] 00020000:00200000:1.0:1789760789.290391:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28831744 chunk: [28831744, 28835840) 28835840, 4096 00020000:00200000:1.0:1789760789.290394:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.290408:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7039, 7039] 00000008:00010000:1.0:1789760789.290421:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.290425:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760789.290433:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28831744, 28835840) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.290443:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28831744=0x1b7f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.290559:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.290562:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28835840 00000020:00010000:1.0:1789760789.290570:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.290574:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.290577:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.290580:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.290583:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.290589:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.290594:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.290609:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760789.290686:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 28831744<>28835840 00000100:00100000:0.0:1789760789.291300:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512765056:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760789.324366:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974358 gen 3 00000008:00010000:0.0:1789760789.324403:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760789.324409:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.324417:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.324434:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760789.324452:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512765056:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760789.324531:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7039, 7039] 00000080:00200000:1.0:1789760789.324541:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.324545:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.324555:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28835840 i_blocks 57351 00000020:00200000:1.0:1789760789.324557:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.324563:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.324586:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28835840, count: 4096 00000080:00200000:1.0:1789760789.324591:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28835840, bytes: 4096 00000080:00200000:1.0:1789760789.324597:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.324603:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760789.324604:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28835840, 28839936) 0 0 00000080:00200000:1.0:1789760789.324608:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.324618:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7040, 7040] 00020000:00200000:1.0:1789760789.324624:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28835840 chunk: [28835840, 28839936) 28839936, 4096 00020000:00200000:1.0:1789760789.324626:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.324638:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7040, 7040] 00000008:00010000:1.0:1789760789.324649:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.324653:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760789.324660:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28835840, 28839936) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.324668:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28835840=0x1b80000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.324711:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.324713:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28839936 00000020:00010000:1.0:1789760789.324718:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.324721:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.324723:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.324725:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.324727:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.324731:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.324735:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.324747:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760789.324821:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 28835840<>28839936 00000100:00100000:0.0:1789760789.324932:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512765184:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760789.357713:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760789.357725:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.357735:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.357815:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760789.357846:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512765184:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760789.360289:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7040, 7040] 00000080:00200000:1.0:1789760789.360310:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760789.360316:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760789.360330:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28839936 i_blocks 57351 00000020:00200000:1.0:1789760789.360335:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760789.360348:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760789.360373:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28839936, count: 4096 00000080:00200000:1.0:1789760789.360379:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28839936, bytes: 4096 00000080:00200000:1.0:1789760789.360391:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760789.360398:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760789.360400:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28839936, 28844032) 0 0 00000080:00200000:1.0:1789760789.360404:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760789.360417:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7041, 7041] 00020000:00200000:1.0:1789760789.360423:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28839936 chunk: [28839936, 28844032) 28844032, 4096 00020000:00200000:1.0:1789760789.360427:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760789.360442:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7041, 7041] 00000008:00010000:1.0:1789760789.360456:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760789.360460:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760789.360469:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28839936, 28844032) append 0 append lockpos 0 00000080:00200000:1.0:1789760789.360481:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28839936=0x1b81000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760789.360532:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760789.360535:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28844032 00000020:00010000:1.0:1789760789.360541:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760789.360544:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.360724:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760789.360729:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760789.360731:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760789.360801:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760789.360809:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760789.360833:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760789.361040:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97480 - object 0x240000400:5 offset 28839936<>28844032 00000100:00100000:0.0:1789760789.361103:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512765312:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760789.431370:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974360 gen 3 00000008:00010000:0.0:1789760789.431418:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760789.431424:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.431433:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.431456:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760789.431480:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512765312:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760789.431573:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7041, 7041] 00000080:00200000:3.0:1789760789.431585:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760789.431590:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760789.431602:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28844032 i_blocks 57351 00000020:00200000:3.0:1789760789.431606:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760789.431614:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760789.431641:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28844032, count: 4096 00000080:00200000:3.0:1789760789.431647:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28844032, bytes: 4096 00000080:00200000:3.0:1789760789.431656:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760789.431664:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760789.431665:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28844032, 28848128) 0 0 00000080:00200000:3.0:1789760789.431669:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760789.431682:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7042, 7042] 00020000:00200000:3.0:1789760789.431688:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28844032 chunk: [28844032, 28848128) 28848128, 4096 00020000:00200000:3.0:1789760789.431692:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760789.431718:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7042, 7042] 00000008:00010000:3.0:1789760789.431732:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760789.431745:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760789.431756:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28844032, 28848128) append 0 append lockpos 0 00000080:00200000:3.0:1789760789.431768:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28844032=0x1b82000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760789.431813:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760789.431816:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28848128 00000020:00010000:3.0:1789760789.431823:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760789.431826:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.431829:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.431832:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760789.431834:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760789.431840:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760789.431846:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760789.431859:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760789.431969:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af800 - object 0x240000400:5 offset 28844032<>28848128 00000100:00100000:3.0:1789760789.432005:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512765440:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760789.484666:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974361 gen 3 00000008:00010000:3.0:1789760789.484700:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760789.484715:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.484724:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.484744:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760789.484768:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512765440:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760789.485215:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7042, 7042] 00000080:00200000:0.0:1789760789.485228:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.485233:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.485246:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28848128 i_blocks 57351 00000020:00200000:0.0:1789760789.485249:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.485257:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.485283:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28848128, count: 4096 00000080:00200000:0.0:1789760789.485288:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28848128, bytes: 4096 00000080:00200000:0.0:1789760789.485295:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.485302:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760789.485304:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28848128, 28852224) 0 0 00000080:00200000:0.0:1789760789.485308:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.485321:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7043, 7043] 00020000:00200000:0.0:1789760789.485329:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28848128 chunk: [28848128, 28852224) 28852224, 4096 00020000:00200000:0.0:1789760789.485332:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.485347:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7043, 7043] 00000008:00010000:0.0:1789760789.485360:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.485365:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760789.485373:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28848128, 28852224) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.485383:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28848128=0x1b83000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.485480:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.485483:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28852224 00000020:00010000:0.0:1789760789.485490:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.485494:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.485497:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.485500:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.485503:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.485509:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.485515:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.485528:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc600 00000008:00100000:0.0:1789760789.485602:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149180 - object 0x240000400:5 offset 28848128<>28852224 00000100:00100000:0.0:1789760789.485714:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760789.485729:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512765568:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760789.536761:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:0.0:1789760789.536774:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760789.536784:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760789.536810:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:0.0:1789760789.536835:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512765568:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760789.537218:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7043, 7043] 00000080:00200000:3.0:1789760789.537239:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760789.537244:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760789.537256:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28852224 i_blocks 57351 00000020:00200000:3.0:1789760789.537260:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760789.537269:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760789.537296:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28852224, count: 4096 00000080:00200000:3.0:1789760789.537302:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28852224, bytes: 4096 00000080:00200000:3.0:1789760789.537312:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760789.537320:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760789.537322:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28852224, 28856320) 0 0 00000080:00200000:3.0:1789760789.537331:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760789.537345:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7044, 7044] 00020000:00200000:3.0:1789760789.537352:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28852224 chunk: [28852224, 28856320) 28856320, 4096 00020000:00200000:3.0:1789760789.537356:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760789.537372:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7044, 7044] 00000008:00010000:3.0:1789760789.537385:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760789.537389:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760789.537398:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28852224, 28856320) append 0 append lockpos 0 00000080:00200000:3.0:1789760789.537409:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28852224=0x1b84000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760789.537452:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760789.537455:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28856320 00000020:00010000:3.0:1789760789.537462:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760789.537465:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.537468:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760789.537471:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760789.537473:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760789.537478:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760789.537484:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760789.537499:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760789.537754:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af800 - object 0x240000400:5 offset 28852224<>28856320 00000100:00100000:3.0:1789760789.537800:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760789.537816:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512765696:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760789.576578:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974363 gen 3 00000008:00010000:3.0:1789760789.576624:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760789.576631:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.576640:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.576657:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760789.576680:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512765696:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760789.576930:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7044, 7044] 00000080:00200000:2.0:1789760789.576944:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760789.576949:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760789.576961:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28856320 i_blocks 57351 00000020:00200000:2.0:1789760789.576964:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760789.576972:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760789.576999:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28856320, count: 4096 00000080:00200000:2.0:1789760789.577006:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28856320, bytes: 4096 00000080:00200000:2.0:1789760789.577014:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760789.577021:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760789.577023:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28856320, 28860416) 0 0 00000080:00200000:2.0:1789760789.577028:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760789.577041:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7045, 7045] 00020000:00200000:2.0:1789760789.577049:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28856320 chunk: [28856320, 28860416) 28860416, 4096 00020000:00200000:2.0:1789760789.577052:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760789.577082:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7045, 7045] 00000008:00010000:2.0:1789760789.577096:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760789.577102:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760789.577111:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28856320, 28860416) append 0 append lockpos 0 00000080:00200000:2.0:1789760789.577122:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28856320=0x1b85000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760789.577178:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760789.577180:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28860416 00000020:00010000:2.0:1789760789.577187:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760789.577191:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.577194:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.577197:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760789.577200:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760789.577206:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760789.577212:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760789.577226:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760789.577306:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 28856320<>28860416 00000100:00100000:2.0:1789760789.577346:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760789.577360:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512765824:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760789.624373:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974364 gen 3 00000008:00010000:3.0:1789760789.624411:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760789.624419:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.624426:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.624442:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760789.624475:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512765824:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760789.624891:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7045, 7045] 00000080:00200000:2.0:1789760789.624929:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760789.624935:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760789.624947:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28860416 i_blocks 57351 00000020:00200000:2.0:1789760789.624951:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760789.624960:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760789.624985:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28860416, count: 4096 00000080:00200000:2.0:1789760789.624992:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28860416, bytes: 4096 00000080:00200000:2.0:1789760789.625000:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760789.625007:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760789.625009:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28860416, 28864512) 0 0 00000080:00200000:2.0:1789760789.625013:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760789.625027:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7046, 7046] 00020000:00200000:2.0:1789760789.625034:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28860416 chunk: [28860416, 28864512) 28864512, 4096 00020000:00200000:2.0:1789760789.625037:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760789.625054:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7046, 7046] 00000008:00010000:2.0:1789760789.625079:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760789.625085:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760789.625096:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28860416, 28864512) append 0 append lockpos 0 00000080:00200000:2.0:1789760789.625108:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28860416=0x1b86000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760789.625153:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760789.625156:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28864512 00000020:00010000:2.0:1789760789.625173:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760789.625176:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.625179:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.625182:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760789.625185:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760789.625191:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760789.625196:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760789.625209:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760789.625287:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 28860416<>28864512 00000100:00100000:3.0:1789760789.625439:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760789.625466:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512765952:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760789.680356:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974365 gen 3 00000008:00010000:3.0:1789760789.680396:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760789.680403:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.680411:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.680428:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760789.680461:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512765952:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760789.680592:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7046, 7046] 00000080:00200000:2.0:1789760789.680603:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760789.680608:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760789.680621:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28864512 i_blocks 57351 00000020:00200000:2.0:1789760789.680624:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760789.680632:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760789.680657:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28864512, count: 4096 00000080:00200000:2.0:1789760789.680664:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28864512, bytes: 4096 00000080:00200000:2.0:1789760789.680671:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760789.680677:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760789.680679:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28864512, 28868608) 0 0 00000080:00200000:2.0:1789760789.680682:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760789.680695:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7047, 7047] 00020000:00200000:2.0:1789760789.680702:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28864512 chunk: [28864512, 28868608) 28868608, 4096 00020000:00200000:2.0:1789760789.680705:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760789.680719:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7047, 7047] 00000008:00010000:2.0:1789760789.680733:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760789.680808:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760789.680819:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28864512, 28868608) append 0 append lockpos 0 00000080:00200000:2.0:1789760789.680830:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28864512=0x1b87000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760789.680872:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760789.680875:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28868608 00000020:00010000:2.0:1789760789.680881:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760789.680884:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.680887:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.680890:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760789.680894:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760789.681040:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760789.681047:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760789.681060:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760789.681145:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 28864512<>28868608 00000100:00100000:3.0:1789760789.681880:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512766080:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760789.725115:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974366 gen 3 00000008:00010000:3.0:1789760789.725149:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760789.725156:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.725174:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.725189:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760789.725211:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512766080:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760789.725344:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7047, 7047] 00000080:00200000:2.0:1789760789.725356:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760789.725361:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760789.725373:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28868608 i_blocks 57351 00000020:00200000:2.0:1789760789.725377:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760789.725384:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760789.725409:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28868608, count: 4096 00000080:00200000:2.0:1789760789.725416:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28868608, bytes: 4096 00000080:00200000:2.0:1789760789.725423:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760789.725431:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760789.725433:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28868608, 28872704) 0 0 00000080:00200000:2.0:1789760789.725437:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760789.725449:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7048, 7048] 00020000:00200000:2.0:1789760789.725456:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28868608 chunk: [28868608, 28872704) 28872704, 4096 00020000:00200000:2.0:1789760789.725521:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760789.725536:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7048, 7048] 00000008:00010000:2.0:1789760789.725550:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760789.725555:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760789.725564:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28868608, 28872704) append 0 append lockpos 0 00000080:00200000:2.0:1789760789.725574:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28868608=0x1b88000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760789.725616:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760789.725619:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28872704 00000020:00010000:2.0:1789760789.725625:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760789.725628:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.725631:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.725634:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760789.725636:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760789.725643:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760789.725648:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760789.725661:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760789.725734:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 28868608<>28872704 00000100:00100000:3.0:1789760789.725823:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512766208:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760789.763344:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760789.763353:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.763359:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.763372:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000080:00200000:2.0:1789760789.763394:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7048, 7048] 00000100:00100000:3.0:1789760789.763395:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512766208:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760789.763407:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760789.763412:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760789.763423:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28872704 i_blocks 57351 00000020:00200000:2.0:1789760789.763426:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760789.763434:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760789.763459:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28872704, count: 4096 00000080:00200000:2.0:1789760789.763683:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28872704, bytes: 4096 00000080:00200000:2.0:1789760789.763691:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760789.763698:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760789.763701:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28872704, 28876800) 0 0 00000080:00200000:2.0:1789760789.763705:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760789.763718:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7049, 7049] 00020000:00200000:2.0:1789760789.763725:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28872704 chunk: [28872704, 28876800) 28876800, 4096 00020000:00200000:2.0:1789760789.763729:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760789.763856:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7049, 7049] 00000008:00010000:2.0:1789760789.763868:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760789.763872:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760789.763879:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28872704, 28876800) append 0 append lockpos 0 00000080:00200000:2.0:1789760789.763887:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28872704=0x1b89000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760789.763947:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760789.763950:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28876800 00000020:00010000:2.0:1789760789.763956:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760789.763959:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.763962:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.763965:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760789.763967:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760789.763973:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760789.763977:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760789.763988:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760789.764048:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 28872704<>28876800 00000100:00100000:3.0:1789760789.764131:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512766336:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760789.802450:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974368 gen 3 00000008:00010000:3.0:1789760789.802557:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760789.802563:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760789.802571:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760789.802624:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760789.802659:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512766336:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760789.802836:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7049, 7049] 00000080:00200000:2.0:1789760789.802849:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760789.802854:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760789.802867:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28876800 i_blocks 57351 00000020:00200000:2.0:1789760789.802870:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760789.802878:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760789.802998:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28876800, count: 4096 00000080:00200000:2.0:1789760789.803006:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28876800, bytes: 4096 00000080:00200000:2.0:1789760789.803014:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760789.803021:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760789.803023:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28876800, 28880896) 0 0 00000080:00200000:2.0:1789760789.803027:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760789.803040:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7050, 7050] 00020000:00200000:2.0:1789760789.803047:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28876800 chunk: [28876800, 28880896) 28880896, 4096 00020000:00200000:2.0:1789760789.803051:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760789.803076:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7050, 7050] 00000008:00010000:2.0:1789760789.803091:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760789.803095:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760789.803104:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28876800, 28880896) append 0 append lockpos 0 00000080:00200000:2.0:1789760789.803114:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28876800=0x1b8a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760789.803230:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760789.803233:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28880896 00000020:00010000:2.0:1789760789.803240:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760789.803243:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.803246:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760789.803250:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760789.803253:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760789.803259:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760789.803264:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760789.803279:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760789.803616:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 28876800<>28880896 00000100:00100000:3.0:1789760789.803845:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512766464:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760789.834034:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974369 gen 3 00000008:00010000:2.0:1789760789.834219:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:2.0:1789760789.834226:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760789.834235:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760789.834296:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:2.0:1789760789.834325:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512766464:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760789.834399:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7050, 7050] 00000080:00200000:0.0:1789760789.834412:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.834417:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.834429:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28880896 i_blocks 57351 00000020:00200000:0.0:1789760789.834433:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.834441:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.834571:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28880896, count: 4096 00000080:00200000:0.0:1789760789.834578:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28880896, bytes: 4096 00000080:00200000:0.0:1789760789.834586:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.834593:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760789.834595:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28880896, 28884992) 0 0 00000080:00200000:0.0:1789760789.834599:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.834613:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7051, 7051] 00020000:00200000:0.0:1789760789.834619:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28880896 chunk: [28880896, 28884992) 28884992, 4096 00020000:00200000:0.0:1789760789.834623:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.834714:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7051, 7051] 00000008:00010000:0.0:1789760789.834730:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.834843:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760789.834854:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28880896, 28884992) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.834866:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28880896=0x1b8b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.834932:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.834935:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28884992 00000020:00010000:0.0:1789760789.834942:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.834945:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.834948:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.834952:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.834955:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.834961:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.834966:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.834980:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc600 00000008:00100000:0.0:1789760789.835056:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b480 - object 0x240000400:5 offset 28880896<>28884992 00000100:00100000:1.0:1789760789.835228:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512766592:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760789.877157:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974370 gen 3 00000008:00010000:1.0:1789760789.877193:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760789.877199:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.877207:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.877216:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760789.877238:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512766592:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760789.877342:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7051, 7051] 00000080:00200000:0.0:1789760789.877355:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.877360:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.877372:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28884992 i_blocks 57351 00000020:00200000:0.0:1789760789.877375:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.877385:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.877409:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28884992, count: 4096 00000080:00200000:0.0:1789760789.877414:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28884992, bytes: 4096 00000080:00200000:0.0:1789760789.877421:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.877427:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760789.877429:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28884992, 28889088) 0 0 00000080:00200000:0.0:1789760789.877433:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.877444:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7052, 7052] 00020000:00200000:0.0:1789760789.877451:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28884992 chunk: [28884992, 28889088) 28889088, 4096 00020000:00200000:0.0:1789760789.877455:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.877470:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7052, 7052] 00000008:00010000:0.0:1789760789.877483:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.877487:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760789.877496:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28884992, 28889088) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.877506:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28884992=0x1b8c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.877549:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.877552:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28889088 00000020:00010000:0.0:1789760789.877558:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.877561:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.877563:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.877566:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.877568:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.877574:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.877580:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.877593:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000008:00100000:0.0:1789760789.877667:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b800 - object 0x240000400:5 offset 28884992<>28889088 00000100:00100000:1.0:1789760789.877753:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760789.877767:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512766720:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760789.903614:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760789.903625:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.903635:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.903655:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760789.903677:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512766720:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760789.903883:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7052, 7052] 00000080:00200000:0.0:1789760789.903926:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.903932:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.903943:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28889088 i_blocks 57351 00000020:00200000:0.0:1789760789.903947:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.903955:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.903986:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28889088, count: 4096 00000080:00200000:0.0:1789760789.903991:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28889088, bytes: 4096 00000080:00200000:0.0:1789760789.903997:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.904002:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760789.904003:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28889088, 28893184) 0 0 00000080:00200000:0.0:1789760789.904007:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.904018:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7053, 7053] 00020000:00200000:0.0:1789760789.904024:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28889088 chunk: [28889088, 28893184) 28893184, 4096 00020000:00200000:0.0:1789760789.904027:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.904041:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7053, 7053] 00000008:00010000:0.0:1789760789.904054:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.904058:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760789.904083:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28889088, 28893184) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.904094:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28889088=0x1b8d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.904138:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.904141:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28893184 00000020:00010000:0.0:1789760789.904147:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.904150:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.904153:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.904156:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.904159:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.904164:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.904170:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.904184:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cccc0 00000008:00100000:0.0:1789760789.904257:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148380 - object 0x240000400:5 offset 28889088<>28893184 00000100:00100000:1.0:1789760789.904387:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760789.904401:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512766848:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760789.934190:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974372 gen 3 00000008:00010000:1.0:1789760789.934237:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760789.934243:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.934251:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.934270:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760789.934291:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512766848:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760789.934544:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7053, 7053] 00000080:00200000:0.0:1789760789.934559:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.934564:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.934575:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28893184 i_blocks 57351 00000020:00200000:0.0:1789760789.934578:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.934585:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.934611:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28893184, count: 4096 00000080:00200000:0.0:1789760789.934618:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28893184, bytes: 4096 00000080:00200000:0.0:1789760789.934661:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.934670:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760789.934672:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28893184, 28897280) 0 0 00000080:00200000:0.0:1789760789.934676:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.934688:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7054, 7054] 00020000:00200000:0.0:1789760789.934694:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28893184 chunk: [28893184, 28897280) 28897280, 4096 00020000:00200000:0.0:1789760789.934699:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.934785:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7054, 7054] 00000008:00010000:0.0:1789760789.934799:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.934804:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760789.934813:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28893184, 28897280) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.934824:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28893184=0x1b8e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.934867:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.934870:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28897280 00000020:00010000:0.0:1789760789.934876:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.934879:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.934882:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.934886:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.934889:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.934895:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.934979:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.934993:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc900 00000008:00100000:0.0:1789760789.935144:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149880 - object 0x240000400:5 offset 28893184<>28897280 00000100:00100000:1.0:1789760789.935197:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512766976:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760789.981948:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760789.981959:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760789.981968:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760789.981990:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760789.982012:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512766976:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760789.982107:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7054, 7054] 00000080:00200000:0.0:1789760789.982120:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760789.982125:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760789.982136:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28897280 i_blocks 57351 00000020:00200000:0.0:1789760789.982139:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760789.982157:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760789.982182:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28897280, count: 4096 00000080:00200000:0.0:1789760789.982188:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28897280, bytes: 4096 00000080:00200000:0.0:1789760789.982195:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760789.982202:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760789.982204:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28897280, 28901376) 0 0 00000080:00200000:0.0:1789760789.982209:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760789.982221:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7055, 7055] 00020000:00200000:0.0:1789760789.982228:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28897280 chunk: [28897280, 28901376) 28901376, 4096 00020000:00200000:0.0:1789760789.982231:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760789.982246:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7055, 7055] 00000008:00010000:0.0:1789760789.982258:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760789.982262:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760789.982272:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28897280, 28901376) append 0 append lockpos 0 00000080:00200000:0.0:1789760789.982282:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28897280=0x1b8f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760789.982327:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760789.982330:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28901376 00000020:00010000:0.0:1789760789.982336:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760789.982339:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.982342:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760789.982345:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760789.982348:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760789.982354:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760789.982359:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760789.982372:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000008:00100000:0.0:1789760789.982450:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a300 - object 0x240000400:5 offset 28897280<>28901376 00000100:00100000:1.0:1789760789.982658:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512767104:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760790.018078:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974374 gen 3 00000008:00010000:1.0:1789760790.018120:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760790.018126:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.018134:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.018150:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760790.018170:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512767104:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760790.019504:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7055, 7055] 00000080:00200000:0.0:1789760790.019521:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.019526:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.019538:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28901376 i_blocks 57351 00000020:00200000:0.0:1789760790.019541:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.019551:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.019575:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28901376, count: 4096 00000080:00200000:0.0:1789760790.019581:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28901376, bytes: 4096 00000080:00200000:0.0:1789760790.019589:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.019597:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.019598:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28901376, 28905472) 0 0 00000080:00200000:0.0:1789760790.019602:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.019614:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7056, 7056] 00020000:00200000:0.0:1789760790.019620:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28901376 chunk: [28901376, 28905472) 28905472, 4096 00020000:00200000:0.0:1789760790.019623:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.019637:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7056, 7056] 00000008:00010000:0.0:1789760790.019650:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.019654:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760790.019663:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28901376, 28905472) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.019676:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28901376=0x1b90000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.019735:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.019738:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28905472 00000020:00010000:0.0:1789760790.019744:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.019747:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.019750:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.019753:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.019755:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.019762:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.019767:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.019783:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccb40 00000008:00100000:0.0:1789760790.019858:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a680 - object 0x240000400:5 offset 28901376<>28905472 00000100:00100000:1.0:1789760790.020039:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760790.020058:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512767232:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760790.078400:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974375 gen 3 00000008:00010000:1.0:1789760790.078440:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760790.078446:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.078467:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.078485:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760790.078507:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512767232:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760790.078795:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7056, 7056] 00000080:00200000:0.0:1789760790.078809:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.078814:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.078825:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28905472 i_blocks 57351 00000020:00200000:0.0:1789760790.078829:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.078839:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.078866:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28905472, count: 4096 00000080:00200000:0.0:1789760790.078871:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28905472, bytes: 4096 00000080:00200000:0.0:1789760790.078879:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.078885:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.078887:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28905472, 28909568) 0 0 00000080:00200000:0.0:1789760790.078890:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.078934:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7057, 7057] 00020000:00200000:0.0:1789760790.078942:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28905472 chunk: [28905472, 28909568) 28909568, 4096 00020000:00200000:0.0:1789760790.078945:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.079034:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7057, 7057] 00000008:00010000:0.0:1789760790.079048:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.079053:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760790.079071:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28905472, 28909568) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.079090:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28905472=0x1b91000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.079131:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.079134:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28909568 00000020:00010000:0.0:1789760790.079141:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.079143:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.079146:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.079149:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.079151:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.079157:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.079162:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.079174:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc300 00000008:00100000:0.0:1789760790.079254:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148e00 - object 0x240000400:5 offset 28905472<>28909568 00000100:00100000:1.0:1789760790.081346:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760790.081373:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512767360:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760790.124668:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974376 gen 3 00000008:00010000:1.0:1789760790.124702:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760790.124709:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.124975:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.125001:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760790.125026:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512767360:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760790.126413:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7057, 7057] 00000080:00200000:0.0:1789760790.126429:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.126434:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.126446:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28909568 i_blocks 57351 00000020:00200000:0.0:1789760790.126450:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.126460:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.126483:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28909568, count: 4096 00000080:00200000:0.0:1789760790.126489:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28909568, bytes: 4096 00000080:00200000:0.0:1789760790.126498:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.126506:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.126508:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28909568, 28913664) 0 0 00000080:00200000:0.0:1789760790.126511:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.126522:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7058, 7058] 00020000:00200000:0.0:1789760790.126528:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28909568 chunk: [28909568, 28913664) 28913664, 4096 00020000:00200000:0.0:1789760790.126532:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.126546:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7058, 7058] 00000008:00010000:0.0:1789760790.126559:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.126563:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760790.126572:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28909568, 28913664) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.126581:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28909568=0x1b92000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.126629:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.126632:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28913664 00000020:00010000:0.0:1789760790.126638:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.126641:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.126644:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.126647:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.126650:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.126656:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.126662:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.126678:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc240 00000008:00100000:0.0:1789760790.127889:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514ad80 - object 0x240000400:5 offset 28909568<>28913664 00000100:00100000:1.0:1789760790.127997:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760790.128018:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512767488:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760790.166525:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760790.166539:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.166550:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.166572:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760790.166802:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512767488:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760790.166996:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7058, 7058] 00000080:00200000:0.0:1789760790.167009:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.167014:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.167026:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28913664 i_blocks 57351 00000020:00200000:0.0:1789760790.167029:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.167037:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.167072:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28913664, count: 4096 00000080:00200000:0.0:1789760790.167078:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28913664, bytes: 4096 00000080:00200000:0.0:1789760790.167086:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.167093:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.167095:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28913664, 28917760) 0 0 00000080:00200000:0.0:1789760790.167099:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.167112:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7059, 7059] 00020000:00200000:0.0:1789760790.167118:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28913664 chunk: [28913664, 28917760) 28917760, 4096 00020000:00200000:0.0:1789760790.167122:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.167135:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7059, 7059] 00000008:00010000:0.0:1789760790.167149:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.167153:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760790.167162:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28913664, 28917760) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.167173:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28913664=0x1b93000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.167375:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.167379:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28917760 00000020:00010000:0.0:1789760790.167387:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.167390:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.167392:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.167395:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.167397:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.167404:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.167410:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.167425:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc480 00000008:00100000:0.0:1789760790.167501:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148700 - object 0x240000400:5 offset 28913664<>28917760 00000100:00100000:1.0:1789760790.167733:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512767616:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760790.216817:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974378 gen 3 00000008:00010000:1.0:1789760790.216881:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760790.216889:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.216936:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.216955:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760790.216982:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512767616:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760790.217310:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7059, 7059] 00000080:00200000:0.0:1789760790.217328:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.217334:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.217348:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28917760 i_blocks 57351 00000020:00200000:0.0:1789760790.217352:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.217364:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.217390:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28917760, count: 4096 00000080:00200000:0.0:1789760790.217397:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28917760, bytes: 4096 00000080:00200000:0.0:1789760790.217408:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.217416:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.217418:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28917760, 28921856) 0 0 00000080:00200000:0.0:1789760790.217423:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.217436:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7060, 7060] 00020000:00200000:0.0:1789760790.217442:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28917760 chunk: [28917760, 28921856) 28921856, 4096 00020000:00200000:0.0:1789760790.217447:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.217462:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7060, 7060] 00000008:00010000:0.0:1789760790.217476:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.217481:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760790.217489:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28917760, 28921856) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.217500:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28917760=0x1b94000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.217549:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.217552:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28921856 00000020:00010000:0.0:1789760790.217558:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.217561:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.217564:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.217567:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.217569:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.217576:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.217582:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.217599:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc840 00000008:00100000:0.0:1789760790.218174:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148a80 - object 0x240000400:5 offset 28917760<>28921856 00000100:00100000:1.0:1789760790.218431:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512767744:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760790.259157:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974379 gen 3 00000008:00010000:1.0:1789760790.259195:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760790.259201:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.259209:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.259286:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760790.259311:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512767744:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760790.262044:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7060, 7060] 00000080:00200000:0.0:1789760790.262077:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760790.262083:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760790.262095:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28921856 i_blocks 57351 00000020:00200000:0.0:1789760790.262100:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760790.262110:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760790.262135:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28921856, count: 4096 00000080:00200000:0.0:1789760790.262142:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28921856, bytes: 4096 00000080:00200000:0.0:1789760790.262150:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760790.262157:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760790.262158:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28921856, 28925952) 0 0 00000080:00200000:0.0:1789760790.262163:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760790.262175:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7061, 7061] 00020000:00200000:0.0:1789760790.262182:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28921856 chunk: [28921856, 28925952) 28925952, 4096 00020000:00200000:0.0:1789760790.262186:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760790.262201:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7061, 7061] 00000008:00010000:0.0:1789760790.262215:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760790.262220:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760790.262229:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28921856, 28925952) append 0 append lockpos 0 00000080:00200000:0.0:1789760790.262239:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28921856=0x1b95000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760790.262291:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760790.262294:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28925952 00000020:00010000:0.0:1789760790.262301:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760790.262304:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.262307:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760790.262309:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760790.262312:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760790.262318:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760790.262323:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760790.262346:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc840 00000008:00100000:0.0:1789760790.263155:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514aa00 - object 0x240000400:5 offset 28921856<>28925952 00000100:00100000:1.0:1789760790.263230:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512767872:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760790.302883:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760790.303745:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512768128:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760790.303823:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2361:1876699512768384:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760790.303855:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1789760790.303861:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e6cf7267-9b19-462a-a263-14115df15dab:2361:1876699512768000:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760790.303889:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512768256:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760790.313293:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:e6cf7267-9b19-462a-a263-14115df15dab:2361:1876699512768000:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760790.313315:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512768256:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760790.313326:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512768128:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760790.313336:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2361:1876699512768384:192.168.203.156@tcp:400:kworker.0 00000100:00100000:1.0:1789760790.322185:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974380 gen 3 00000008:00010000:1.0:1789760790.322220:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc840 00000020:00200000:1.0:1789760790.322226:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760790.322234:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760790.322253:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc840 00000100:00100000:1.0:1789760790.322275:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512767872:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760790.322363:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7061, 7061] 00000080:00200000:2.0:1789760790.322376:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.322381:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.322393:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28925952 i_blocks 57351 00000020:00200000:2.0:1789760790.322396:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.322404:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.322431:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28925952, count: 4096 00000080:00200000:2.0:1789760790.322437:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28925952, bytes: 4096 00000080:00200000:2.0:1789760790.322444:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.322505:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.322507:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28925952, 28930048) 0 0 00000080:00200000:2.0:1789760790.322512:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.322525:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7062, 7062] 00020000:00200000:2.0:1789760790.322532:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28925952 chunk: [28925952, 28930048) 28930048, 4096 00020000:00200000:2.0:1789760790.322535:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.322550:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7062, 7062] 00000008:00010000:2.0:1789760790.322564:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.322569:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760790.322577:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28925952, 28930048) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.322589:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28925952=0x1b96000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.322632:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.322635:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28930048 00000020:00010000:2.0:1789760790.322642:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.322644:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.322647:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.322650:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.322652:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.322658:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.322664:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.322677:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760790.322751:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 28925952<>28930048 00000100:00100000:3.0:1789760790.322888:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512768512:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760790.358405:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974381 gen 3 00000008:00010000:3.0:1789760790.358442:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760790.358447:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.358455:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.358473:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760790.358496:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512768512:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760790.358583:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7062, 7062] 00000080:00200000:2.0:1789760790.358595:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.358600:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.358611:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28930048 i_blocks 57351 00000020:00200000:2.0:1789760790.358615:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.358622:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.358646:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28930048, count: 4096 00000080:00200000:2.0:1789760790.358652:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28930048, bytes: 4096 00000080:00200000:2.0:1789760790.358660:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.358667:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.358669:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28930048, 28934144) 0 0 00000080:00200000:2.0:1789760790.358672:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.358685:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7063, 7063] 00020000:00200000:2.0:1789760790.358692:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28930048 chunk: [28930048, 28934144) 28934144, 4096 00020000:00200000:2.0:1789760790.358695:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.358708:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7063, 7063] 00000008:00010000:2.0:1789760790.358721:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.358725:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760790.358734:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28930048, 28934144) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.358744:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28930048=0x1b97000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.358786:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.358789:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28934144 00000020:00010000:2.0:1789760790.358795:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.358798:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.358801:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.358804:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.358806:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.358812:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.358817:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.358830:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760790.358934:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 28930048<>28934144 00000100:00100000:3.0:1789760790.359255:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760790.359269:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512768640:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760790.399628:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760790.399639:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.399648:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.399707:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760790.399731:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512768640:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760790.400128:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7063, 7063] 00000080:00200000:2.0:1789760790.400141:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.400146:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.400895:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28934144 i_blocks 57351 00000020:00200000:2.0:1789760790.400933:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.400943:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.400970:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28934144, count: 4096 00000080:00200000:2.0:1789760790.400977:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28934144, bytes: 4096 00000080:00200000:2.0:1789760790.400986:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.400992:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.400993:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28934144, 28938240) 0 0 00000080:00200000:2.0:1789760790.400996:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.401007:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7064, 7064] 00020000:00200000:2.0:1789760790.401013:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28934144 chunk: [28934144, 28938240) 28938240, 4096 00020000:00200000:2.0:1789760790.401015:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.401029:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7064, 7064] 00000008:00010000:2.0:1789760790.401041:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.401044:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760790.401051:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28934144, 28938240) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.401071:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28934144=0x1b98000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.401129:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.401132:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28938240 00000020:00010000:2.0:1789760790.401139:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.401142:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.401145:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.401148:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.401150:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.401157:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.401163:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.401181:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760790.401262:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 28934144<>28938240 00000100:00100000:3.0:1789760790.401360:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512768768:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760790.457952:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974382 gen 3 00000100:00100000:3.0:1789760790.457986:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368c380 x1876699512768768/t4294974383(4294974383) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760806 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760790.458021:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760790.458027:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.458034:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.458153:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760790.458174:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512768768:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760790.458276:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7064, 7064] 00000080:00200000:2.0:1789760790.458290:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.458294:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.458305:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28938240 i_blocks 57351 00000020:00200000:2.0:1789760790.458308:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.458315:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.458338:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28938240, count: 4096 00000080:00200000:2.0:1789760790.458343:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28938240, bytes: 4096 00000080:00200000:2.0:1789760790.458350:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.458357:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.458358:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28938240, 28942336) 0 0 00000080:00200000:2.0:1789760790.458362:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.458374:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7065, 7065] 00020000:00200000:2.0:1789760790.458381:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28938240 chunk: [28938240, 28942336) 28942336, 4096 00020000:00200000:2.0:1789760790.458385:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.458400:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7065, 7065] 00000008:00010000:2.0:1789760790.458413:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.458418:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760790.458427:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28938240, 28942336) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.458438:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28938240=0x1b99000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.458485:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.458488:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28942336 00000020:00010000:2.0:1789760790.458496:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.458499:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.458502:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.458505:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.458508:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.458514:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.458520:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.458534:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760790.458606:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 28938240<>28942336 00000100:00100000:3.0:1789760790.458667:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512768896:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760790.513282:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974384 gen 3 00000008:00010000:3.0:1789760790.513331:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760790.513338:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.513346:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.513364:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760790.513388:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512768896:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760790.513619:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7065, 7065] 00000080:00200000:2.0:1789760790.513631:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.513637:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.513647:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28942336 i_blocks 57351 00000020:00200000:2.0:1789760790.513650:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.513658:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.513683:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28942336, count: 4096 00000080:00200000:2.0:1789760790.513689:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28942336, bytes: 4096 00000080:00200000:2.0:1789760790.513695:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.513702:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.513704:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28942336, 28946432) 0 0 00000080:00200000:2.0:1789760790.513707:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.513720:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7066, 7066] 00020000:00200000:2.0:1789760790.513725:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28942336 chunk: [28942336, 28946432) 28946432, 4096 00020000:00200000:2.0:1789760790.513729:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.513744:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7066, 7066] 00000008:00010000:2.0:1789760790.513757:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.513761:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760790.513771:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28942336, 28946432) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.513781:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28942336=0x1b9a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.513824:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.513827:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28946432 00000020:00010000:2.0:1789760790.513833:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.513836:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.513839:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.513842:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.513844:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.513851:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.513856:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.513870:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760790.513984:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 28942336<>28946432 00000100:00100000:3.0:1789760790.514139:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760790.514153:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512769024:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760790.560015:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974385 gen 3 00000008:00010000:3.0:1789760790.560050:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760790.560065:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.560075:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.560093:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760790.560118:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512769024:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760790.560210:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7066, 7066] 00000080:00200000:2.0:1789760790.560222:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.560227:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.560239:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28946432 i_blocks 57351 00000020:00200000:2.0:1789760790.560243:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.560252:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.560278:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28946432, count: 4096 00000080:00200000:2.0:1789760790.560284:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28946432, bytes: 4096 00000080:00200000:2.0:1789760790.560292:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.560299:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.560301:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28946432, 28950528) 0 0 00000080:00200000:2.0:1789760790.560305:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.560319:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7067, 7067] 00020000:00200000:2.0:1789760790.560328:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28946432 chunk: [28946432, 28950528) 28950528, 4096 00020000:00200000:2.0:1789760790.560332:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.560347:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7067, 7067] 00000008:00010000:2.0:1789760790.560362:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.560366:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760790.560376:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28946432, 28950528) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.560387:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28946432=0x1b9b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.560431:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.560434:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28950528 00000020:00010000:2.0:1789760790.560442:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.560445:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.560459:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.560463:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.560466:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.560472:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.560478:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.560491:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760790.560566:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 28946432<>28950528 00000100:00100000:3.0:1789760790.560709:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512769152:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760790.595437:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760790.595449:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.595457:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.595473:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760790.595498:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512769152:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760790.595788:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7067, 7067] 00000080:00200000:2.0:1789760790.595801:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.595805:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.595817:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28950528 i_blocks 57351 00000020:00200000:2.0:1789760790.595820:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.595827:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.595859:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28950528, count: 4096 00000080:00200000:2.0:1789760790.595865:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28950528, bytes: 4096 00000080:00200000:2.0:1789760790.595875:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.595884:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.595886:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28950528, 28954624) 0 0 00000080:00200000:2.0:1789760790.595890:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.595937:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7068, 7068] 00020000:00200000:2.0:1789760790.595944:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28950528 chunk: [28950528, 28954624) 28954624, 4096 00020000:00200000:2.0:1789760790.595948:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.595966:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7068, 7068] 00000008:00010000:2.0:1789760790.595983:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.595988:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760790.595996:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28950528, 28954624) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.596006:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28950528=0x1b9c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.596069:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.596072:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28954624 00000020:00010000:2.0:1789760790.596078:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.596081:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.596083:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.596086:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.596089:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.596094:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.596100:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.596114:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760790.596189:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 28950528<>28954624 00000100:00100000:3.0:1789760790.596400:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760790.596417:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512769280:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760790.644953:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974387 gen 3 00000008:00010000:3.0:1789760790.645000:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760790.645006:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.645014:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.645032:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760790.645067:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512769280:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760790.646582:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7068, 7068] 00000080:00200000:2.0:1789760790.646598:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.646604:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.646617:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28954624 i_blocks 57351 00000020:00200000:2.0:1789760790.646621:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.646631:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.646656:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28954624, count: 4096 00000080:00200000:2.0:1789760790.646663:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28954624, bytes: 4096 00000080:00200000:2.0:1789760790.646672:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.646680:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.646682:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28954624, 28958720) 0 0 00000080:00200000:2.0:1789760790.646686:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.646699:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7069, 7069] 00020000:00200000:2.0:1789760790.646706:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28954624 chunk: [28954624, 28958720) 28958720, 4096 00020000:00200000:2.0:1789760790.646710:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.646726:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7069, 7069] 00000008:00010000:2.0:1789760790.646740:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.646745:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760790.646755:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28954624, 28958720) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.646767:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28954624=0x1b9d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.646827:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.646830:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28958720 00000020:00010000:2.0:1789760790.646836:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.646840:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.646843:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.646846:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.646848:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.646855:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.646862:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.646878:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760790.647003:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 28954624<>28958720 00000100:00100000:3.0:1789760790.647161:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512769408:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760790.682214:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760790.682225:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.682234:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.682251:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760790.682274:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512769408:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760790.682357:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7069, 7069] 00000080:00200000:2.0:1789760790.682370:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.682375:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.682387:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28958720 i_blocks 57351 00000020:00200000:2.0:1789760790.682390:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.682398:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.682441:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28958720, count: 4096 00000080:00200000:2.0:1789760790.682454:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28958720, bytes: 4096 00000080:00200000:2.0:1789760790.682462:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.682469:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.682471:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28958720, 28962816) 0 0 00000080:00200000:2.0:1789760790.682475:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.682489:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7070, 7070] 00020000:00200000:2.0:1789760790.682496:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28958720 chunk: [28958720, 28962816) 28962816, 4096 00020000:00200000:2.0:1789760790.682500:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.682517:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7070, 7070] 00000008:00010000:2.0:1789760790.682532:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.682536:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760790.682546:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28958720, 28962816) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.682560:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28958720=0x1b9e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.682604:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.682606:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28962816 00000020:00010000:2.0:1789760790.682612:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.682614:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.682616:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.682618:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.682620:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.682625:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.682629:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.682640:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760790.682702:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 28958720<>28962816 00000100:00100000:3.0:1789760790.684565:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760790.684591:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512769536:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760790.731841:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974389 gen 3 00000008:00010000:3.0:1789760790.732292:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760790.732300:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.732308:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.732376:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760790.732401:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512769536:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760790.732655:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7070, 7070] 00000080:00200000:2.0:1789760790.732667:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.732672:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.732683:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28962816 i_blocks 57351 00000020:00200000:2.0:1789760790.732686:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.732694:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.732719:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28962816, count: 4096 00000080:00200000:2.0:1789760790.732725:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28962816, bytes: 4096 00000080:00200000:2.0:1789760790.732734:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.732741:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.732742:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28962816, 28966912) 0 0 00000080:00200000:2.0:1789760790.732746:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.732758:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7071, 7071] 00020000:00200000:2.0:1789760790.732765:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28962816 chunk: [28962816, 28966912) 28966912, 4096 00020000:00200000:2.0:1789760790.732769:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.732782:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7071, 7071] 00000008:00010000:2.0:1789760790.732795:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.732799:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760790.732808:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28962816, 28966912) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.732817:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28962816=0x1b9f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.733151:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.733154:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28966912 00000020:00010000:2.0:1789760790.733162:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.733165:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.733168:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.733175:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.733178:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.733186:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.733192:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.733208:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760790.733281:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 28962816<>28966912 00000100:00100000:3.0:1789760790.733491:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512769664:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760790.775623:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760790.775634:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.775642:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.775662:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760790.775686:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512769664:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760790.775771:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7071, 7071] 00000080:00200000:2.0:1789760790.775794:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.775800:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.775811:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28966912 i_blocks 57351 00000020:00200000:2.0:1789760790.775814:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.775822:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.775848:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28966912, count: 4096 00000080:00200000:2.0:1789760790.775854:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28966912, bytes: 4096 00000080:00200000:2.0:1789760790.775862:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.775869:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.775871:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28966912, 28971008) 0 0 00000080:00200000:2.0:1789760790.775875:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.775888:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7072, 7072] 00020000:00200000:2.0:1789760790.775895:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28966912 chunk: [28966912, 28971008) 28971008, 4096 00020000:00200000:2.0:1789760790.775920:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.775937:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7072, 7072] 00000008:00010000:2.0:1789760790.775951:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.775956:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760790.775965:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28966912, 28971008) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.775976:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28966912=0x1ba0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.776018:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.776021:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28971008 00000020:00010000:2.0:1789760790.776028:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.776031:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.776034:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.776037:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.776039:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.776045:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.776051:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.776073:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760790.776157:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 28966912<>28971008 00000100:00100000:3.0:1789760790.776459:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512769792:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760790.829427:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974391 gen 3 00000008:00010000:3.0:1789760790.829541:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760790.829548:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.829556:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.829610:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760790.829635:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512769792:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760790.829704:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7072, 7072] 00000080:00200000:2.0:1789760790.829717:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.829722:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.829734:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28971008 i_blocks 57351 00000020:00200000:2.0:1789760790.829737:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.829745:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.829770:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28971008, count: 4096 00000080:00200000:2.0:1789760790.829776:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28971008, bytes: 4096 00000080:00200000:2.0:1789760790.829784:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.829791:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.829793:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28971008, 28975104) 0 0 00000080:00200000:2.0:1789760790.829797:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.829809:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7073, 7073] 00020000:00200000:2.0:1789760790.829817:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28971008 chunk: [28971008, 28975104) 28975104, 4096 00020000:00200000:2.0:1789760790.829820:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.829834:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7073, 7073] 00000008:00010000:2.0:1789760790.829848:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.829853:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760790.829861:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28971008, 28975104) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.829871:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28971008=0x1ba1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.830014:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.830017:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28975104 00000020:00010000:2.0:1789760790.830024:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.830028:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.830223:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.830227:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.830229:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.830237:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.830243:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.830258:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760790.830333:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 28971008<>28975104 00000100:00100000:3.0:1789760790.830393:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512769920:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760790.871866:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760790.871878:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.871886:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.871918:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760790.871941:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512769920:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760790.872097:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7073, 7073] 00000080:00200000:2.0:1789760790.872111:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.872117:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.872130:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28975104 i_blocks 57351 00000020:00200000:2.0:1789760790.872133:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.872142:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.872183:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28975104, count: 4096 00000080:00200000:2.0:1789760790.872190:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28975104, bytes: 4096 00000080:00200000:2.0:1789760790.872199:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.872206:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.872209:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28975104, 28979200) 0 0 00000080:00200000:2.0:1789760790.872213:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.872227:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7074, 7074] 00020000:00200000:2.0:1789760790.872233:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28975104 chunk: [28975104, 28979200) 28979200, 4096 00020000:00200000:2.0:1789760790.872237:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.872252:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7074, 7074] 00000008:00010000:2.0:1789760790.872266:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.872271:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760790.872279:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28975104, 28979200) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.872290:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28975104=0x1ba2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.872332:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.872335:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28979200 00000020:00010000:2.0:1789760790.872341:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.872345:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.872347:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.872350:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.872353:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.872359:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.872364:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.872378:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760790.872453:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 28975104<>28979200 00000100:00100000:2.0:1789760790.872493:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512770048:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760790.903733:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974393 gen 3 00000008:00010000:3.0:1789760790.903850:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760790.903858:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.903866:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.903975:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760790.904004:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512770048:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760790.904502:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7074, 7074] 00000080:00200000:2.0:1789760790.904515:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.904520:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.904531:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28979200 i_blocks 57351 00000020:00200000:2.0:1789760790.904535:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.904543:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.904569:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28979200, count: 4096 00000080:00200000:2.0:1789760790.904576:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28979200, bytes: 4096 00000080:00200000:2.0:1789760790.904583:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.904591:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.904593:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28979200, 28983296) 0 0 00000080:00200000:2.0:1789760790.904596:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.904609:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7075, 7075] 00020000:00200000:2.0:1789760790.904615:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28979200 chunk: [28979200, 28983296) 28983296, 4096 00020000:00200000:2.0:1789760790.904619:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.904632:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7075, 7075] 00000008:00010000:2.0:1789760790.904646:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.904651:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760790.904661:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28979200, 28983296) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.904671:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28979200=0x1ba3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.904714:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.904717:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28983296 00000020:00010000:2.0:1789760790.904723:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.904726:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.904729:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.904732:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.904735:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.904741:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.904747:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.904762:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760790.904928:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 28979200<>28983296 00000100:00100000:3.0:1789760790.904990:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512770176:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760790.947872:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974394 gen 3 00000008:00010000:3.0:1789760790.947978:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760790.947984:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760790.947993:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760790.948009:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760790.948030:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512770176:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760790.948120:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7075, 7075] 00000080:00200000:2.0:1789760790.948134:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760790.948139:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760790.948149:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28983296 i_blocks 57351 00000020:00200000:2.0:1789760790.948221:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760790.948230:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760790.948258:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28983296, count: 4096 00000080:00200000:2.0:1789760790.948264:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28983296, bytes: 4096 00000080:00200000:2.0:1789760790.948271:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760790.948279:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760790.948281:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28983296, 28987392) 0 0 00000080:00200000:2.0:1789760790.948285:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760790.948297:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7076, 7076] 00020000:00200000:2.0:1789760790.948304:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28983296 chunk: [28983296, 28987392) 28987392, 4096 00020000:00200000:2.0:1789760790.948307:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760790.948321:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7076, 7076] 00000008:00010000:2.0:1789760790.948333:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760790.948347:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760790.948357:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28983296, 28987392) append 0 append lockpos 0 00000080:00200000:2.0:1789760790.948374:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28983296=0x1ba4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760790.948414:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760790.948416:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28987392 00000020:00010000:2.0:1789760790.948422:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760790.948425:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.948427:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760790.948430:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760790.948433:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760790.948438:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760790.948442:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760790.948455:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760790.948526:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 28983296<>28987392 00000100:00100000:3.0:1789760790.948730:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512770304:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760791.002958:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974395 gen 3 00000008:00010000:3.0:1789760791.002994:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760791.003000:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.003020:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.003093:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760791.003118:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512770304:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760791.006413:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7076, 7076] 00000080:00200000:2.0:1789760791.006541:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760791.006548:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760791.006562:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28987392 i_blocks 57351 00000020:00200000:2.0:1789760791.006568:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760791.006579:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760791.006607:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28987392, count: 4096 00000080:00200000:2.0:1789760791.006614:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28987392, bytes: 4096 00000080:00200000:2.0:1789760791.006625:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760791.006633:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760791.006635:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28987392, 28991488) 0 0 00000080:00200000:2.0:1789760791.006639:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760791.006652:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7077, 7077] 00020000:00200000:2.0:1789760791.006659:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28987392 chunk: [28987392, 28991488) 28991488, 4096 00020000:00200000:2.0:1789760791.006663:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760791.006678:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7077, 7077] 00000008:00010000:2.0:1789760791.006694:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760791.006698:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760791.006713:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28987392, 28991488) append 0 append lockpos 0 00000080:00200000:2.0:1789760791.006726:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28987392=0x1ba5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760791.006793:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760791.006795:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28991488 00000020:00010000:2.0:1789760791.006802:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760791.006805:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.006809:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.006822:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760791.006824:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760791.006831:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760791.006837:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760791.006862:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760791.006974:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 28987392<>28991488 00000100:00100000:3.0:1789760791.007082:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512770432:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760791.057811:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760791.057822:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.057832:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.057884:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760791.057999:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512770432:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760791.058496:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7077, 7077] 00000080:00200000:2.0:1789760791.058754:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760791.058759:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760791.058771:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28991488 i_blocks 57351 00000020:00200000:2.0:1789760791.058776:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760791.058785:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760791.059022:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28991488, count: 4096 00000080:00200000:2.0:1789760791.059030:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28991488, bytes: 4096 00000080:00200000:2.0:1789760791.059039:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760791.059064:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760791.059067:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28991488, 28995584) 0 0 00000080:00200000:2.0:1789760791.059071:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760791.059084:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7078, 7078] 00020000:00200000:2.0:1789760791.059091:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28991488 chunk: [28991488, 28995584) 28995584, 4096 00020000:00200000:2.0:1789760791.059095:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760791.059109:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7078, 7078] 00000008:00010000:2.0:1789760791.059123:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760791.059127:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760791.059138:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28991488, 28995584) append 0 append lockpos 0 00000080:00200000:2.0:1789760791.059149:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28991488=0x1ba6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760791.059196:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760791.059198:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28995584 00000020:00010000:2.0:1789760791.059205:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760791.059208:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.059210:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.059213:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760791.059215:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760791.059222:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760791.059228:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760791.059245:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760791.059778:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 28991488<>28995584 00000100:00100000:3.0:1789760791.060191:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512770560:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760791.117473:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974397 gen 3 00000008:00010000:3.0:1789760791.117527:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760791.117533:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.117541:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.117677:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760791.117701:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512770560:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760791.117811:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7078, 7078] 00000080:00200000:2.0:1789760791.117824:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760791.117829:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760791.117839:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28995584 i_blocks 57351 00000020:00200000:2.0:1789760791.117843:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760791.117850:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760791.117874:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28995584, count: 4096 00000080:00200000:2.0:1789760791.117880:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28995584, bytes: 4096 00000080:00200000:2.0:1789760791.117887:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760791.117894:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760791.117896:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28995584, 28999680) 0 0 00000080:00200000:2.0:1789760791.118018:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760791.118032:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7079, 7079] 00020000:00200000:2.0:1789760791.118040:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28995584 chunk: [28995584, 28999680) 28999680, 4096 00020000:00200000:2.0:1789760791.118043:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760791.118093:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7079, 7079] 00000008:00010000:2.0:1789760791.118109:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760791.118114:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760791.118123:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28995584, 28999680) append 0 append lockpos 0 00000080:00200000:2.0:1789760791.118133:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28995584=0x1ba7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760791.118175:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760791.118177:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 28999680 00000020:00010000:2.0:1789760791.118184:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760791.118186:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.118189:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.118192:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760791.118195:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760791.118201:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760791.118206:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760791.118219:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760791.118394:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 28995584<>28999680 00000100:00100000:3.0:1789760791.118512:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512770688:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760791.171826:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974398 gen 3 00000008:00010000:3.0:1789760791.171863:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760791.171870:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.171878:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.171898:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760791.171945:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512770688:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760791.172032:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7079, 7079] 00000080:00200000:2.0:1789760791.172044:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760791.172049:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760791.172072:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 28999680 i_blocks 57351 00000020:00200000:2.0:1789760791.172076:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760791.172083:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760791.172109:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 28999680, count: 4096 00000080:00200000:2.0:1789760791.172115:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 28999680, bytes: 4096 00000080:00200000:2.0:1789760791.172123:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760791.172130:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760791.172132:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [28999680, 29003776) 0 0 00000080:00200000:2.0:1789760791.172136:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760791.172149:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7080, 7080] 00020000:00200000:2.0:1789760791.172155:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 28999680 chunk: [28999680, 29003776) 29003776, 4096 00020000:00200000:2.0:1789760791.172159:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760791.172174:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7080, 7080] 00000008:00010000:2.0:1789760791.172188:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760791.172192:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760791.172201:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [28999680, 29003776) append 0 append lockpos 0 00000080:00200000:2.0:1789760791.172211:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=28999680=0x1ba8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760791.172254:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760791.172257:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29003776 00000020:00010000:2.0:1789760791.172263:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760791.172266:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.172269:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.172271:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760791.172274:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760791.172280:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760791.172296:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760791.172310:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760791.172384:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 28999680<>29003776 00000100:00100000:3.0:1789760791.172680:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512770816:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760791.231399:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974399 gen 3 00000008:00010000:3.0:1789760791.231433:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760791.231439:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.231511:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.231529:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760791.231554:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512770816:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760791.231644:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7080, 7080] 00000080:00200000:2.0:1789760791.231658:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760791.231663:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760791.231675:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29003776 i_blocks 57351 00000020:00200000:2.0:1789760791.231678:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760791.231686:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760791.231711:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29003776, count: 4096 00000080:00200000:2.0:1789760791.231717:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29003776, bytes: 4096 00000080:00200000:2.0:1789760791.231736:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760791.231744:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760791.231746:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29003776, 29007872) 0 0 00000080:00200000:2.0:1789760791.231750:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760791.231764:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7081, 7081] 00020000:00200000:2.0:1789760791.231771:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29003776 chunk: [29003776, 29007872) 29007872, 4096 00020000:00200000:2.0:1789760791.231775:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760791.231969:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7081, 7081] 00000008:00010000:2.0:1789760791.231985:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760791.231990:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760791.231999:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29003776, 29007872) append 0 append lockpos 0 00000080:00200000:2.0:1789760791.232009:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29003776=0x1ba9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760791.232063:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760791.232066:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29007872 00000020:00010000:2.0:1789760791.232073:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760791.232076:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.232078:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.232080:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760791.232083:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760791.232089:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760791.232095:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760791.232108:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760791.232179:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 29003776<>29007872 00000100:00100000:3.0:1789760791.232273:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512770944:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760791.274148:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760791.274160:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.274169:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.274187:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760791.274212:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512770944:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760791.274286:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7081, 7081] 00000080:00200000:2.0:1789760791.274299:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760791.274304:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760791.274316:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29007872 i_blocks 57351 00000020:00200000:2.0:1789760791.274319:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760791.274327:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760791.274351:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29007872, count: 4096 00000080:00200000:2.0:1789760791.274357:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29007872, bytes: 4096 00000080:00200000:2.0:1789760791.274364:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760791.274371:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760791.274373:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29007872, 29011968) 0 0 00000080:00200000:2.0:1789760791.274377:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760791.274390:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7082, 7082] 00020000:00200000:2.0:1789760791.274398:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29007872 chunk: [29007872, 29011968) 29011968, 4096 00020000:00200000:2.0:1789760791.274402:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760791.274416:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7082, 7082] 00000008:00010000:2.0:1789760791.274429:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760791.274434:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760791.274556:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29007872, 29011968) append 0 append lockpos 0 00000080:00200000:2.0:1789760791.274569:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29007872=0x1baa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760791.274613:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760791.274616:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29011968 00000020:00010000:2.0:1789760791.274623:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760791.274626:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.274629:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.274632:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760791.274635:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760791.274642:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760791.274647:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760791.274661:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760791.274733:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 29007872<>29011968 00000100:00100000:3.0:1789760791.274934:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512771072:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760791.326030:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760791.327393:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974401 gen 3 00000008:00010000:3.0:1789760791.327436:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760791.327453:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.327461:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.327480:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760791.327500:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512771072:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760791.327541:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7082, 7082] 00000080:00200000:0.0:1789760791.327622:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.327627:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.327639:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29011968 i_blocks 57351 00000020:00200000:0.0:1789760791.327643:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.327651:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.327683:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29011968, count: 4096 00000080:00200000:0.0:1789760791.327689:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29011968, bytes: 4096 00000080:00200000:0.0:1789760791.327696:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.327702:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.327703:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29011968, 29016064) 0 0 00000080:00200000:0.0:1789760791.327708:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.327720:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7083, 7083] 00020000:00200000:0.0:1789760791.327727:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29011968 chunk: [29011968, 29016064) 29016064, 4096 00020000:00200000:0.0:1789760791.327731:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.327746:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7083, 7083] 00000008:00010000:0.0:1789760791.327758:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.327763:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760791.327771:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29011968, 29016064) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.327781:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29011968=0x1bab000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.327823:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.327826:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29016064 00000020:00010000:0.0:1789760791.327833:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.327836:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.327839:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.327842:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.327845:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.327851:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.327856:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.327868:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc480 00000008:00100000:0.0:1789760791.327966:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514aa00 - object 0x240000400:5 offset 29011968<>29016064 00000100:00100000:1.0:1789760791.328028:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760791.328534:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512771200:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760791.375627:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974402 gen 3 00000008:00010000:1.0:1789760791.375663:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc480 00000020:00200000:1.0:1789760791.375670:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.375680:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.375700:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc480 00000100:00100000:1.0:1789760791.375724:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512771200:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760791.376109:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7083, 7083] 00000080:00200000:2.0:1789760791.376121:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760791.376126:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760791.376138:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29016064 i_blocks 57351 00000020:00200000:2.0:1789760791.376142:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760791.376150:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760791.376178:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29016064, count: 4096 00000080:00200000:2.0:1789760791.376184:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29016064, bytes: 4096 00000080:00200000:2.0:1789760791.376191:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760791.376198:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760791.376200:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29016064, 29020160) 0 0 00000080:00200000:2.0:1789760791.376203:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760791.376216:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7084, 7084] 00020000:00200000:2.0:1789760791.376222:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29016064 chunk: [29016064, 29020160) 29020160, 4096 00020000:00200000:2.0:1789760791.376226:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760791.376241:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7084, 7084] 00000008:00010000:2.0:1789760791.376256:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760791.376261:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760791.376270:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29016064, 29020160) append 0 append lockpos 0 00000080:00200000:2.0:1789760791.376281:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29016064=0x1bac000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760791.376326:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760791.376328:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29020160 00000020:00010000:2.0:1789760791.376335:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760791.376338:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.376341:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.376344:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760791.376347:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760791.376353:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760791.376359:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760791.376374:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760791.376464:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 29016064<>29020160 00000100:00100000:3.0:1789760791.376564:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760791.376580:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512771328:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760791.430303:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760791.430313:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.430322:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.430341:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760791.430367:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512771328:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760791.430445:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7084, 7084] 00000080:00200000:2.0:1789760791.430458:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760791.430463:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760791.430483:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29020160 i_blocks 57351 00000020:00200000:2.0:1789760791.430487:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760791.430495:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760791.430519:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29020160, count: 4096 00000080:00200000:2.0:1789760791.430526:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29020160, bytes: 4096 00000080:00200000:2.0:1789760791.430533:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760791.430540:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760791.430542:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29020160, 29024256) 0 0 00000080:00200000:2.0:1789760791.430546:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760791.430559:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7085, 7085] 00020000:00200000:2.0:1789760791.430565:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29020160 chunk: [29020160, 29024256) 29024256, 4096 00020000:00200000:2.0:1789760791.430569:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760791.430584:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7085, 7085] 00000008:00010000:2.0:1789760791.430598:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760791.430603:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760791.430611:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29020160, 29024256) append 0 append lockpos 0 00000080:00200000:2.0:1789760791.430621:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29020160=0x1bad000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760791.430664:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760791.430666:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29024256 00000020:00010000:2.0:1789760791.430673:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760791.430676:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.430680:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.430683:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760791.430686:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760791.430693:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760791.430698:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760791.430710:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760791.430783:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 29020160<>29024256 00000100:00100000:3.0:1789760791.430922:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760791.430936:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512771456:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760791.499279:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974404 gen 3 00000008:00010000:3.0:1789760791.499326:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760791.499333:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.499340:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.499359:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760791.499382:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512771456:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760791.499545:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7085, 7085] 00000080:00200000:2.0:1789760791.499559:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760791.499563:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760791.499575:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29024256 i_blocks 57351 00000020:00200000:2.0:1789760791.499578:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760791.499585:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760791.499610:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29024256, count: 4096 00000080:00200000:2.0:1789760791.499616:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29024256, bytes: 4096 00000080:00200000:2.0:1789760791.499623:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760791.499630:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760791.499632:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29024256, 29028352) 0 0 00000080:00200000:2.0:1789760791.499636:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760791.499648:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7086, 7086] 00020000:00200000:2.0:1789760791.499654:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29024256 chunk: [29024256, 29028352) 29028352, 4096 00020000:00200000:2.0:1789760791.499657:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760791.499671:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7086, 7086] 00000008:00010000:2.0:1789760791.499684:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760791.499689:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760791.499698:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29024256, 29028352) append 0 append lockpos 0 00000080:00200000:2.0:1789760791.499708:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29024256=0x1bae000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760791.499756:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760791.499758:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29028352 00000020:00010000:2.0:1789760791.499765:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760791.499769:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.499771:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.499775:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760791.499790:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760791.499797:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760791.499803:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760791.499816:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760791.499886:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 29024256<>29028352 00000100:00100000:3.0:1789760791.500059:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760791.500073:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512771584:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760791.533642:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760791.533653:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.533662:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.533875:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760791.533928:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512771584:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760791.534042:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7086, 7086] 00000080:00200000:2.0:1789760791.534065:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760791.534069:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760791.534081:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29028352 i_blocks 57351 00000020:00200000:2.0:1789760791.534084:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760791.534091:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760791.534117:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29028352, count: 4096 00000080:00200000:2.0:1789760791.534123:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29028352, bytes: 4096 00000080:00200000:2.0:1789760791.534130:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760791.534138:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760791.534140:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29028352, 29032448) 0 0 00000080:00200000:2.0:1789760791.534143:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760791.534155:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7087, 7087] 00020000:00200000:2.0:1789760791.534161:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29028352 chunk: [29028352, 29032448) 29032448, 4096 00020000:00200000:2.0:1789760791.534164:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760791.534178:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7087, 7087] 00000008:00010000:2.0:1789760791.534190:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760791.534194:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760791.534202:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29028352, 29032448) append 0 append lockpos 0 00000080:00200000:2.0:1789760791.534212:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29028352=0x1baf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760791.534248:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760791.534251:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29032448 00000020:00010000:2.0:1789760791.534256:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760791.534259:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.534261:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.534264:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760791.534267:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760791.534272:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760791.534277:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760791.534290:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760791.534364:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 29028352<>29032448 00000100:00100000:3.0:1789760791.534417:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512771712:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760791.566151:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974406 gen 3 00000008:00010000:3.0:1789760791.566196:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760791.566202:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.566209:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.566227:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760791.566253:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512771712:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760791.566341:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7087, 7087] 00000080:00200000:2.0:1789760791.566355:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760791.566360:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760791.566372:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29032448 i_blocks 57351 00000020:00200000:2.0:1789760791.566375:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760791.566383:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760791.566410:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29032448, count: 4096 00000080:00200000:2.0:1789760791.566415:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29032448, bytes: 4096 00000080:00200000:2.0:1789760791.566421:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760791.566428:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760791.566430:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29032448, 29036544) 0 0 00000080:00200000:2.0:1789760791.566434:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760791.566456:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7088, 7088] 00020000:00200000:2.0:1789760791.566465:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29032448 chunk: [29032448, 29036544) 29036544, 4096 00020000:00200000:2.0:1789760791.566469:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760791.566484:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7088, 7088] 00000008:00010000:2.0:1789760791.566499:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760791.566504:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760791.566513:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29032448, 29036544) append 0 append lockpos 0 00000080:00200000:2.0:1789760791.566523:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29032448=0x1bb0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760791.566565:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760791.566568:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29036544 00000020:00010000:2.0:1789760791.566574:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760791.566577:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.566580:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.566582:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760791.566585:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760791.566590:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760791.566596:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760791.566610:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760791.566685:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 29032448<>29036544 00000100:00100000:3.0:1789760791.567015:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512771840:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760791.599466:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974407 gen 3 00000008:00010000:3.0:1789760791.599510:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760791.599517:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.599526:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.599579:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760791.599603:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512771840:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760791.600184:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7088, 7088] 00000080:00200000:2.0:1789760791.600200:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760791.600205:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760791.600218:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29036544 i_blocks 57351 00000020:00200000:2.0:1789760791.600222:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760791.600232:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760791.600268:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29036544, count: 4096 00000080:00200000:2.0:1789760791.600274:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29036544, bytes: 4096 00000080:00200000:2.0:1789760791.600284:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760791.600292:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760791.600294:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29036544, 29040640) 0 0 00000080:00200000:2.0:1789760791.600298:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760791.600315:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7089, 7089] 00020000:00200000:2.0:1789760791.600323:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29036544 chunk: [29036544, 29040640) 29040640, 4096 00020000:00200000:2.0:1789760791.600327:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760791.600345:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7089, 7089] 00000008:00010000:2.0:1789760791.600361:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760791.600368:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760791.600376:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29036544, 29040640) append 0 append lockpos 0 00000080:00200000:2.0:1789760791.600389:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29036544=0x1bb1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760791.600585:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760791.600588:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29040640 00000020:00010000:2.0:1789760791.600596:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760791.600599:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.600603:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760791.600606:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760791.600609:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760791.600616:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760791.600622:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760791.600643:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760791.601040:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 29036544<>29040640 00000100:00100000:2.0:1789760791.601245:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760791.601262:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512771968:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760791.633751:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974408 gen 3 00000008:00010000:3.0:1789760791.633802:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760791.633809:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760791.633818:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760791.633835:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760791.633860:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512771968:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760791.634082:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7089, 7089] 00000080:00200000:1.0:1789760791.634094:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760791.634099:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760791.634109:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29040640 i_blocks 57351 00000020:00200000:1.0:1789760791.634112:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760791.634120:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760791.634143:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29040640, count: 4096 00000080:00200000:1.0:1789760791.634149:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29040640, bytes: 4096 00000080:00200000:1.0:1789760791.634156:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760791.634162:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760791.634164:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29040640, 29044736) 0 0 00000080:00200000:1.0:1789760791.634167:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760791.634180:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7090, 7090] 00020000:00200000:1.0:1789760791.634187:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29040640 chunk: [29040640, 29044736) 29044736, 4096 00020000:00200000:1.0:1789760791.634190:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760791.634206:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7090, 7090] 00000008:00010000:1.0:1789760791.634218:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760791.634223:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760791.634232:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29040640, 29044736) append 0 append lockpos 0 00000080:00200000:1.0:1789760791.634243:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29040640=0x1bb2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760791.634286:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760791.634289:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29044736 00000020:00010000:1.0:1789760791.634296:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760791.634299:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.634302:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.634305:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760791.634307:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760791.634313:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760791.634319:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760791.634332:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760791.634409:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96680 - object 0x240000400:5 offset 29040640<>29044736 00000100:00100000:0.0:1789760791.634567:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760791.634582:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512772096:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760791.667306:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974409 gen 3 00000008:00010000:0.0:1789760791.667342:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760791.667349:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760791.667357:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760791.667377:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760791.667399:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512772096:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760791.667549:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7090, 7090] 00000080:00200000:1.0:1789760791.667562:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760791.667568:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760791.667580:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29044736 i_blocks 57351 00000020:00200000:1.0:1789760791.667583:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760791.667592:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760791.667620:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29044736, count: 4096 00000080:00200000:1.0:1789760791.667626:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29044736, bytes: 4096 00000080:00200000:1.0:1789760791.667634:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760791.667641:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760791.667643:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29044736, 29048832) 0 0 00000080:00200000:1.0:1789760791.667647:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760791.667658:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7091, 7091] 00020000:00200000:1.0:1789760791.667663:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29044736 chunk: [29044736, 29048832) 29048832, 4096 00020000:00200000:1.0:1789760791.667666:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760791.667679:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7091, 7091] 00000008:00010000:1.0:1789760791.667691:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760791.667695:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760791.667704:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29044736, 29048832) append 0 append lockpos 0 00000080:00200000:1.0:1789760791.667713:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29044736=0x1bb3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760791.667751:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760791.667754:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29048832 00000020:00010000:1.0:1789760791.667760:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760791.667763:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.667766:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.667769:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760791.667772:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760791.667891:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760791.667898:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760791.667948:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760791.668025:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95880 - object 0x240000400:5 offset 29044736<>29048832 00000100:00100000:0.0:1789760791.668118:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512772224:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760791.701984:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760791.701997:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760791.702007:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760791.702028:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760791.702063:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512772224:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760791.702215:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7091, 7091] 00000080:00200000:1.0:1789760791.702228:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760791.702233:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760791.702245:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29048832 i_blocks 57351 00000020:00200000:1.0:1789760791.702248:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760791.702257:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760791.702282:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29048832, count: 4096 00000080:00200000:1.0:1789760791.702288:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29048832, bytes: 4096 00000080:00200000:1.0:1789760791.702296:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760791.702303:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760791.702305:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29048832, 29052928) 0 0 00000080:00200000:1.0:1789760791.702308:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760791.702321:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7092, 7092] 00020000:00200000:1.0:1789760791.702328:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29048832 chunk: [29048832, 29052928) 29052928, 4096 00020000:00200000:1.0:1789760791.702331:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760791.702347:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7092, 7092] 00000008:00010000:1.0:1789760791.702360:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760791.702365:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760791.702373:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29048832, 29052928) append 0 append lockpos 0 00000080:00200000:1.0:1789760791.702384:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29048832=0x1bb4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760791.702428:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760791.702431:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29052928 00000020:00010000:1.0:1789760791.702437:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760791.702452:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.702455:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.702458:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760791.702460:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760791.702466:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760791.702472:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760791.702487:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760791.702566:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96d80 - object 0x240000400:5 offset 29048832<>29052928 00000100:00100000:0.0:1789760791.702725:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760791.702741:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512772352:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760791.738077:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974411 gen 3 00000008:00010000:0.0:1789760791.738127:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760791.738134:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760791.738148:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760791.738158:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760791.738180:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512772352:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760791.738275:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7092, 7092] 00000080:00200000:1.0:1789760791.738287:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760791.738292:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760791.738304:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29052928 i_blocks 57351 00000020:00200000:1.0:1789760791.738307:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760791.738316:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760791.738344:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29052928, count: 4096 00000080:00200000:1.0:1789760791.738351:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29052928, bytes: 4096 00000080:00200000:1.0:1789760791.738358:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760791.738366:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760791.738368:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29052928, 29057024) 0 0 00000080:00200000:1.0:1789760791.738372:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760791.738384:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7093, 7093] 00020000:00200000:1.0:1789760791.738391:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29052928 chunk: [29052928, 29057024) 29057024, 4096 00020000:00200000:1.0:1789760791.738395:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760791.738409:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7093, 7093] 00000008:00010000:1.0:1789760791.738421:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760791.738426:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760791.738435:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29052928, 29057024) append 0 append lockpos 0 00000080:00200000:1.0:1789760791.738445:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29052928=0x1bb5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760791.738531:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760791.738535:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29057024 00000020:00010000:1.0:1789760791.738543:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760791.738546:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.738549:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.738552:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760791.738555:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760791.738561:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760791.738566:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760791.738580:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760791.738657:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96d80 - object 0x240000400:5 offset 29052928<>29057024 00000100:00100000:0.0:1789760791.738712:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512772480:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760791.768250:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974412 gen 3 00000008:00010000:0.0:1789760791.768423:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760791.768429:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760791.768438:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760791.768457:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760791.768615:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512772480:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760791.769114:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7093, 7093] 00000080:00200000:1.0:1789760791.769129:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760791.769134:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760791.769146:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29057024 i_blocks 57351 00000020:00200000:1.0:1789760791.769149:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760791.769158:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760791.769185:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29057024, count: 4096 00000080:00200000:1.0:1789760791.769191:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29057024, bytes: 4096 00000080:00200000:1.0:1789760791.769206:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760791.769214:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760791.769216:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29057024, 29061120) 0 0 00000080:00200000:1.0:1789760791.769220:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760791.769233:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7094, 7094] 00020000:00200000:1.0:1789760791.769240:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29057024 chunk: [29057024, 29061120) 29061120, 4096 00020000:00200000:1.0:1789760791.769243:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760791.769260:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7094, 7094] 00000008:00010000:1.0:1789760791.769274:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760791.769279:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760791.769288:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29057024, 29061120) append 0 append lockpos 0 00000080:00200000:1.0:1789760791.769300:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29057024=0x1bb6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760791.769351:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760791.769354:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29061120 00000020:00010000:1.0:1789760791.769362:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760791.769366:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.769369:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.769372:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760791.769374:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760791.769381:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760791.769387:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760791.769403:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760791.769545:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c95880 - object 0x240000400:5 offset 29057024<>29061120 00000100:00100000:0.0:1789760791.769654:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760791.769670:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512772608:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760791.814629:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974413 gen 3 00000008:00010000:0.0:1789760791.814665:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760791.814671:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760791.814679:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760791.814807:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760791.814832:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512772608:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760791.817593:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7094, 7094] 00000080:00200000:1.0:1789760791.817610:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760791.817615:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760791.817627:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29061120 i_blocks 57351 00000020:00200000:1.0:1789760791.817631:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760791.817644:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760791.817669:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29061120, count: 4096 00000080:00200000:1.0:1789760791.817676:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29061120, bytes: 4096 00000080:00200000:1.0:1789760791.817686:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760791.817694:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760791.817696:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29061120, 29065216) 0 0 00000080:00200000:1.0:1789760791.817700:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760791.817714:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7095, 7095] 00020000:00200000:1.0:1789760791.817781:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29061120 chunk: [29061120, 29065216) 29065216, 4096 00020000:00200000:1.0:1789760791.817785:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760791.817804:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7095, 7095] 00000008:00010000:1.0:1789760791.817819:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760791.817824:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760791.817834:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29061120, 29065216) append 0 append lockpos 0 00000080:00200000:1.0:1789760791.817845:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29061120=0x1bb7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760791.817897:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760791.817915:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29065216 00000020:00010000:1.0:1789760791.817922:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760791.817925:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.817928:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.817931:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760791.817934:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760791.817941:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760791.817947:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760791.817971:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760791.818271:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c96680 - object 0x240000400:5 offset 29061120<>29065216 00000100:00100000:0.0:1789760791.818344:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512772736:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760791.895073:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974414 gen 3 00000008:00010000:0.0:1789760791.895117:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760791.895124:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760791.895133:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760791.895150:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760791.895170:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512772736:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760791.895867:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7095, 7095] 00000080:00200000:1.0:1789760791.895881:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760791.895886:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760791.895938:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29065216 i_blocks 57351 00000020:00200000:1.0:1789760791.896633:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760791.896649:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760791.896677:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29065216, count: 4096 00000080:00200000:1.0:1789760791.896684:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29065216, bytes: 4096 00000080:00200000:1.0:1789760791.896695:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760791.896701:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760791.896703:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29065216, 29069312) 0 0 00000080:00200000:1.0:1789760791.896707:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760791.896721:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7096, 7096] 00020000:00200000:1.0:1789760791.896728:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29065216 chunk: [29065216, 29069312) 29069312, 4096 00020000:00200000:1.0:1789760791.896732:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760791.896748:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7096, 7096] 00000008:00010000:1.0:1789760791.896763:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760791.896841:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760791.896853:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29065216, 29069312) append 0 append lockpos 0 00000080:00200000:1.0:1789760791.896865:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29065216=0x1bb8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760791.897386:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760791.897390:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29069312 00000020:00010000:1.0:1789760791.897398:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760791.897401:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.897404:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760791.897408:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760791.897411:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760791.897418:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760791.897424:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760791.897443:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760791.897538:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97480 - object 0x240000400:5 offset 29065216<>29069312 00000100:00100000:1.0:1789760791.898694:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760791.898716:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512772864:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760791.949468:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974415 gen 3 00000008:00010000:1.0:1789760791.949510:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760791.949516:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760791.949539:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760791.949559:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760791.949583:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512772864:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760791.949781:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7096, 7096] 00000080:00200000:0.0:1789760791.949794:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760791.949800:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760791.949812:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29069312 i_blocks 57351 00000020:00200000:0.0:1789760791.949815:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760791.949824:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760791.949850:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29069312, count: 4096 00000080:00200000:0.0:1789760791.949856:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29069312, bytes: 4096 00000080:00200000:0.0:1789760791.949865:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760791.949872:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760791.949874:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29069312, 29073408) 0 0 00000080:00200000:0.0:1789760791.949878:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760791.950080:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7097, 7097] 00020000:00200000:0.0:1789760791.950088:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29069312 chunk: [29069312, 29073408) 29073408, 4096 00020000:00200000:0.0:1789760791.950092:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760791.950108:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7097, 7097] 00000008:00010000:0.0:1789760791.950121:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760791.950125:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760791.950135:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29069312, 29073408) append 0 append lockpos 0 00000080:00200000:0.0:1789760791.950145:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29069312=0x1bb9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760791.950186:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760791.950189:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29073408 00000020:00010000:0.0:1789760791.950196:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760791.950199:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.950202:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760791.950206:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760791.950208:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760791.950215:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760791.950220:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760791.950235:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc240 00000008:00100000:0.0:1789760791.950307:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148a80 - object 0x240000400:5 offset 29069312<>29073408 00000100:00100000:0.0:1789760791.950334:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760791.950348:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512772992:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760792.011414:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974416 gen 3 00000008:00010000:0.0:1789760792.011508:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:0.0:1789760792.011515:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.011524:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.011601:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:0.0:1789760792.011627:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512772992:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760792.011822:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7097, 7097] 00000080:00200000:3.0:1789760792.011836:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760792.011842:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760792.011854:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29073408 i_blocks 57351 00000020:00200000:3.0:1789760792.011858:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760792.011866:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760792.012626:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29073408, count: 4096 00000080:00200000:3.0:1789760792.012635:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29073408, bytes: 4096 00000080:00200000:3.0:1789760792.012645:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760792.012652:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760792.012654:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29073408, 29077504) 0 0 00000080:00200000:3.0:1789760792.012658:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760792.012671:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7098, 7098] 00020000:00200000:3.0:1789760792.012677:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29073408 chunk: [29073408, 29077504) 29077504, 4096 00020000:00200000:3.0:1789760792.012681:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760792.012696:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7098, 7098] 00000008:00010000:3.0:1789760792.012709:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760792.012713:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760792.012722:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29073408, 29077504) append 0 append lockpos 0 00000080:00200000:3.0:1789760792.012735:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29073408=0x1bba000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760792.012782:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760792.012784:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29077504 00000020:00010000:3.0:1789760792.012791:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760792.012794:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.012797:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760792.012800:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760792.012803:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760792.012809:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760792.012816:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760792.012833:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760792.013125:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad880 - object 0x240000400:5 offset 29073408<>29077504 00000100:00100000:2.0:1789760792.013181:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760792.013366:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512773120:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760792.078260:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760792.078272:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.078461:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.078483:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760792.078507:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512773120:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760792.078608:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7098, 7098] 00000080:00200000:0.0:1789760792.078620:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.078624:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.078635:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29077504 i_blocks 57351 00000020:00200000:0.0:1789760792.078638:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.078645:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.078764:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29077504, count: 4096 00000080:00200000:0.0:1789760792.078771:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29077504, bytes: 4096 00000080:00200000:0.0:1789760792.078779:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.078786:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760792.078788:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29077504, 29081600) 0 0 00000080:00200000:0.0:1789760792.078792:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.078805:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7099, 7099] 00020000:00200000:0.0:1789760792.078812:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29077504 chunk: [29077504, 29081600) 29081600, 4096 00020000:00200000:0.0:1789760792.078816:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.078830:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7099, 7099] 00000008:00010000:0.0:1789760792.078843:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.078848:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760792.078858:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29077504, 29081600) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.078869:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29077504=0x1bbb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.078990:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.078992:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29081600 00000020:00010000:0.0:1789760792.078999:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.079002:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.079005:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.079008:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.079011:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.079017:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.079021:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.079166:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc240 00000008:00100000:0.0:1789760792.079417:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148a80 - object 0x240000400:5 offset 29077504<>29081600 00000100:00100000:1.0:1789760792.079500:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760792.079518:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512773248:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760792.111056:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974418 gen 3 00000008:00010000:1.0:1789760792.111113:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc240 00000020:00200000:1.0:1789760792.111120:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.111128:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.111149:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc240 00000100:00100000:1.0:1789760792.111171:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512773248:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760792.112475:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7099, 7099] 00000080:00200000:0.0:1789760792.112491:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.112495:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.112507:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29081600 i_blocks 57351 00000020:00200000:0.0:1789760792.112511:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.112520:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.112547:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29081600, count: 4096 00000080:00200000:0.0:1789760792.112552:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29081600, bytes: 4096 00000080:00200000:0.0:1789760792.112560:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.112566:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760792.112568:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29081600, 29085696) 0 0 00000080:00200000:0.0:1789760792.112571:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.112583:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7100, 7100] 00020000:00200000:0.0:1789760792.112589:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29081600 chunk: [29081600, 29085696) 29085696, 4096 00020000:00200000:0.0:1789760792.112593:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.112608:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7100, 7100] 00000008:00010000:0.0:1789760792.112621:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.112626:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760792.112634:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29081600, 29085696) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.112645:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29081600=0x1bbc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.112812:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.112815:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29085696 00000020:00010000:0.0:1789760792.112822:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.112825:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.112828:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.112831:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.112833:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.112840:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.112846:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.112862:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc300 00000008:00100000:0.0:1789760792.113200:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148700 - object 0x240000400:5 offset 29081600<>29085696 00000100:00100000:1.0:1789760792.113258:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512773376:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760792.161460:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974419 gen 3 00000008:00010000:1.0:1789760792.161499:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc300 00000020:00200000:1.0:1789760792.161505:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.161514:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.161536:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc300 00000100:00100000:1.0:1789760792.161558:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512773376:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760792.161656:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7100, 7100] 00000080:00200000:2.0:1789760792.161669:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.161674:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.161685:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29085696 i_blocks 57351 00000020:00200000:2.0:1789760792.161688:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.161696:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.161721:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29085696, count: 4096 00000080:00200000:2.0:1789760792.161738:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29085696, bytes: 4096 00000080:00200000:2.0:1789760792.161745:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.161751:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.161753:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29085696, 29089792) 0 0 00000080:00200000:2.0:1789760792.161756:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.161768:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7101, 7101] 00020000:00200000:2.0:1789760792.161773:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29085696 chunk: [29085696, 29089792) 29089792, 4096 00020000:00200000:2.0:1789760792.161776:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.161789:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7101, 7101] 00000008:00010000:2.0:1789760792.161802:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.161807:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760792.161816:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29085696, 29089792) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.161827:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29085696=0x1bbd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.161866:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.161868:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29089792 00000020:00010000:2.0:1789760792.161874:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.161876:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.161879:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.161881:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.161884:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.161890:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.161897:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.162008:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760792.162096:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 29085696<>29089792 00000100:00100000:3.0:1789760792.162161:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512773504:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760792.189939:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974420 gen 3 00000008:00010000:2.0:1789760792.189973:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:2.0:1789760792.189980:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760792.189987:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760792.190008:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:2.0:1789760792.190031:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512773504:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760792.190363:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7101, 7101] 00000080:00200000:1.0:1789760792.190375:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.190379:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.190390:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29089792 i_blocks 57351 00000020:00200000:1.0:1789760792.190393:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.190401:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.190426:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29089792, count: 4096 00000080:00200000:1.0:1789760792.190431:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29089792, bytes: 4096 00000080:00200000:1.0:1789760792.190449:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.190456:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.190458:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29089792, 29093888) 0 0 00000080:00200000:1.0:1789760792.190461:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.190473:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7102, 7102] 00020000:00200000:1.0:1789760792.190479:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29089792 chunk: [29089792, 29093888) 29093888, 4096 00020000:00200000:1.0:1789760792.190483:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.190496:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7102, 7102] 00000008:00010000:1.0:1789760792.190509:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.190512:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760792.190520:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29089792, 29093888) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.190530:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29089792=0x1bbe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.190568:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.190570:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29093888 00000020:00010000:1.0:1789760792.190576:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.190579:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.190582:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.190585:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.190588:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.190594:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.190599:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.190613:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760792.190687:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97480 - object 0x240000400:5 offset 29089792<>29093888 00000100:00100000:1.0:1789760792.190731:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760792.190743:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512773632:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760792.230050:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974421 gen 3 00000008:00010000:1.0:1789760792.230086:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:1.0:1789760792.230093:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760792.230101:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760792.230121:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:1.0:1789760792.230144:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512773632:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760792.230233:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7102, 7102] 00000080:00200000:2.0:1789760792.230247:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.230252:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.230263:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29093888 i_blocks 57351 00000020:00200000:2.0:1789760792.230267:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.230284:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.230310:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29093888, count: 4096 00000080:00200000:2.0:1789760792.230317:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29093888, bytes: 4096 00000080:00200000:2.0:1789760792.230324:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.230332:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.230334:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29093888, 29097984) 0 0 00000080:00200000:2.0:1789760792.230338:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.230351:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7103, 7103] 00020000:00200000:2.0:1789760792.230359:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29093888 chunk: [29093888, 29097984) 29097984, 4096 00020000:00200000:2.0:1789760792.230362:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.230377:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7103, 7103] 00000008:00010000:2.0:1789760792.230391:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.230396:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760792.230404:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29093888, 29097984) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.230416:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29093888=0x1bbf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.230459:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.230462:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29097984 00000020:00010000:2.0:1789760792.230469:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.230472:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.230474:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.230477:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.230480:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.230487:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.230492:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.230505:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760792.230586:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 29093888<>29097984 00000100:00100000:2.0:1789760792.230639:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760792.230651:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512773760:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760792.267461:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974422 gen 3 00000008:00010000:2.0:1789760792.267500:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:2.0:1789760792.267506:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760792.267515:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760792.267538:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:2.0:1789760792.267564:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512773760:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760792.267828:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7103, 7103] 00000080:00200000:1.0:1789760792.267841:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.267846:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.267857:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29097984 i_blocks 57351 00000020:00200000:1.0:1789760792.267861:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.267869:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.267931:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29097984, count: 4096 00000080:00200000:1.0:1789760792.267938:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29097984, bytes: 4096 00000080:00200000:1.0:1789760792.267945:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.267953:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.267955:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29097984, 29102080) 0 0 00000080:00200000:1.0:1789760792.267959:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.267972:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7104, 7104] 00020000:00200000:1.0:1789760792.267978:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29097984 chunk: [29097984, 29102080) 29102080, 4096 00020000:00200000:1.0:1789760792.267982:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.267997:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7104, 7104] 00000008:00010000:1.0:1789760792.268011:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.268015:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760792.268522:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29097984, 29102080) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.268535:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29097984=0x1bc0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.268583:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.268586:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29102080 00000020:00010000:1.0:1789760792.268593:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.268596:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.268598:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.268754:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.268759:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.268766:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.268773:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.268790:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42780 00000008:00100000:1.0:1789760792.268867:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97480 - object 0x240000400:5 offset 29097984<>29102080 00000100:00100000:0.0:1789760792.268966:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512773888:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760792.310777:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974423 gen 3 00000008:00010000:0.0:1789760792.310825:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42780 00000020:00200000:0.0:1789760792.310833:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.310842:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.310860:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42780 00000100:00100000:0.0:1789760792.310884:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512773888:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760792.311235:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7104, 7104] 00000080:00200000:1.0:1789760792.311249:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.311255:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.311269:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29102080 i_blocks 57351 00000020:00200000:1.0:1789760792.311273:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.311282:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.311312:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29102080, count: 4096 00000080:00200000:1.0:1789760792.311318:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29102080, bytes: 4096 00000080:00200000:1.0:1789760792.311326:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.311334:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.311336:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29102080, 29106176) 0 0 00000080:00200000:1.0:1789760792.311341:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.311354:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7105, 7105] 00020000:00200000:1.0:1789760792.311361:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29102080 chunk: [29102080, 29106176) 29106176, 4096 00020000:00200000:1.0:1789760792.311365:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.311380:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7105, 7105] 00000008:00010000:1.0:1789760792.311394:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.311399:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760792.311408:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29102080, 29106176) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.311420:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29102080=0x1bc1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.311465:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.311468:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29106176 00000020:00010000:1.0:1789760792.311474:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.311478:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.311481:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.311484:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.311487:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.311493:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.311499:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.311514:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760792.311592:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c94e00 - object 0x240000400:5 offset 29102080<>29106176 00000100:00100000:0.0:1789760792.311826:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760792.311844:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512774016:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760792.349741:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760792.360319:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974424 gen 3 00000008:00010000:0.0:1789760792.360360:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760792.360367:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.360377:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.360398:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760792.360420:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512774016:192.168.203.156@tcp:4:dd.0 00000080:00200000:1.0:1789760792.360558:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7105, 7105] 00000080:00200000:1.0:1789760792.360571:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1789760792.360576:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:1.0:1789760792.360588:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29106176 i_blocks 57351 00000020:00200000:1.0:1789760792.360591:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1789760792.360600:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:1.0:1789760792.360627:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29106176, count: 4096 00000080:00200000:1.0:1789760792.360634:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29106176, bytes: 4096 00000080:00200000:1.0:1789760792.360642:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:1.0:1789760792.360649:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:1.0:1789760792.360651:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29106176, 29110272) 0 0 00000080:00200000:1.0:1789760792.360655:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:1.0:1789760792.360668:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7106, 7106] 00020000:00200000:1.0:1789760792.360674:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29106176 chunk: [29106176, 29110272) 29110272, 4096 00020000:00200000:1.0:1789760792.360677:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1789760792.360692:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7106, 7106] 00000008:00010000:1.0:1789760792.360706:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:1.0:1789760792.360711:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018e84db0, osc lock ffff8dc0125f6850, flags 0x0 00000080:00200000:1.0:1789760792.360721:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29106176, 29110272) append 0 append lockpos 0 00000080:00200000:1.0:1789760792.360731:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29106176=0x1bc2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:1.0:1789760792.360776:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1789760792.360779:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29110272 00000020:00010000:1.0:1789760792.360786:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:1.0:1789760792.360789:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.360792:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018e84db0 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:1.0:1789760792.360795:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:1.0:1789760792.360798:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018e84db0 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:1.0:1789760792.360803:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:1.0:1789760792.360809:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:1.0:1789760792.360822:0:252:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc004f42540 00000008:00100000:1.0:1789760792.361219:0:252:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005c97b80 - object 0x240000400:5 offset 29106176<>29110272 00000100:00100000:0.0:1789760792.361278:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512774144:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760792.421512:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974425 gen 3 00000008:00010000:0.0:1789760792.421549:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc004f42540 00000020:00200000:0.0:1789760792.421556:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.421565:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.421591:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc004f42540 00000100:00100000:0.0:1789760792.421616:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512774144:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760792.421775:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7106, 7106] 00000080:00200000:2.0:1789760792.421786:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.421829:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.421840:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29110272 i_blocks 57351 00000020:00200000:2.0:1789760792.421843:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.421849:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.421871:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29110272, count: 4096 00000080:00200000:2.0:1789760792.421875:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29110272, bytes: 4096 00000080:00200000:2.0:1789760792.421881:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.421887:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.421888:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29110272, 29114368) 0 0 00000080:00200000:2.0:1789760792.421892:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.421927:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7107, 7107] 00020000:00200000:2.0:1789760792.421935:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29110272 chunk: [29110272, 29114368) 29114368, 4096 00020000:00200000:2.0:1789760792.421939:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.421954:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7107, 7107] 00000008:00010000:2.0:1789760792.421969:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.421975:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760792.421984:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29110272, 29114368) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.421995:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29110272=0x1bc3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.422039:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.422053:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29114368 00000020:00010000:2.0:1789760792.422061:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.422064:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.422068:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.422071:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.422074:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.422082:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.422105:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.422120:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760792.422201:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 29110272<>29114368 00000100:00100000:3.0:1789760792.422267:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512774272:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760792.464826:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760792.464837:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.464846:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.464871:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760792.464895:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512774272:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760792.465435:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7107, 7107] 00000080:00200000:2.0:1789760792.465448:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.465454:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.465466:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29114368 i_blocks 57351 00000020:00200000:2.0:1789760792.465469:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.465477:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.465503:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29114368, count: 4096 00000080:00200000:2.0:1789760792.465509:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29114368, bytes: 4096 00000080:00200000:2.0:1789760792.465516:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.465523:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.465525:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29114368, 29118464) 0 0 00000080:00200000:2.0:1789760792.465528:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.465540:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7108, 7108] 00020000:00200000:2.0:1789760792.465547:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29114368 chunk: [29114368, 29118464) 29118464, 4096 00020000:00200000:2.0:1789760792.465551:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.465566:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7108, 7108] 00000008:00010000:2.0:1789760792.465579:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.465584:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760792.465593:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29114368, 29118464) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.465604:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29114368=0x1bc4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.465646:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.465649:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29118464 00000020:00010000:2.0:1789760792.465655:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.465658:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.465661:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.465664:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.465667:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.465673:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.465678:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.465693:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760792.465768:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 29114368<>29118464 00000100:00100000:3.0:1789760792.465830:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512774400:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760792.513112:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974426 gen 3 00000100:00100000:3.0:1789760792.513141:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368c700 x1876699512774400/t4294974427(4294974427) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760808 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760792.513178:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760792.513184:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.513191:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.513209:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760792.513228:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512774400:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760792.513350:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7108, 7108] 00000080:00200000:2.0:1789760792.513362:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.513367:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.513379:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29118464 i_blocks 57351 00000020:00200000:2.0:1789760792.513382:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.513390:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.513415:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29118464, count: 4096 00000080:00200000:2.0:1789760792.513421:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29118464, bytes: 4096 00000080:00200000:2.0:1789760792.513428:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.513444:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.513447:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29118464, 29122560) 0 0 00000080:00200000:2.0:1789760792.513450:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.513464:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7109, 7109] 00020000:00200000:2.0:1789760792.513471:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29118464 chunk: [29118464, 29122560) 29122560, 4096 00020000:00200000:2.0:1789760792.513475:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.513489:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7109, 7109] 00000008:00010000:2.0:1789760792.513501:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.513505:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760792.513513:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29118464, 29122560) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.513523:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29118464=0x1bc5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.513567:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.513570:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29122560 00000020:00010000:2.0:1789760792.513576:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.513580:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.513584:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.513587:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.513590:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.513596:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.513601:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.513613:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760792.513682:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 29118464<>29122560 00000100:00100000:3.0:1789760792.513930:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760792.514072:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512774528:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760792.552801:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974428 gen 3 00000008:00010000:3.0:1789760792.552844:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760792.552850:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.552858:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.552877:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760792.552956:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512774528:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760792.553229:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7109, 7109] 00000080:00200000:2.0:1789760792.553243:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.553248:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.553260:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29122560 i_blocks 57351 00000020:00200000:2.0:1789760792.553264:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.553272:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.553299:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29122560, count: 4096 00000080:00200000:2.0:1789760792.553306:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29122560, bytes: 4096 00000080:00200000:2.0:1789760792.553314:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.553321:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.553323:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29122560, 29126656) 0 0 00000080:00200000:2.0:1789760792.553327:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.553341:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7110, 7110] 00020000:00200000:2.0:1789760792.553347:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29122560 chunk: [29122560, 29126656) 29126656, 4096 00020000:00200000:2.0:1789760792.553351:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.553365:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7110, 7110] 00000008:00010000:2.0:1789760792.553379:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.553384:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760792.553393:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29122560, 29126656) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.553404:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29122560=0x1bc6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.553457:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.553460:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29126656 00000020:00010000:2.0:1789760792.553467:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.553470:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.553473:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.553476:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.553478:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.553485:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.553490:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.553503:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760792.553577:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 29122560<>29126656 00000100:00100000:2.0:1789760792.553615:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760792.553629:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512774656:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760792.590355:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974429 gen 3 00000008:00010000:3.0:1789760792.590390:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760792.590396:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.590405:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.590422:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760792.590455:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512774656:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760792.590546:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7110, 7110] 00000080:00200000:2.0:1789760792.590558:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.590563:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.590574:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29126656 i_blocks 57351 00000020:00200000:2.0:1789760792.590578:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.590585:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.590611:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29126656, count: 4096 00000080:00200000:2.0:1789760792.590617:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29126656, bytes: 4096 00000080:00200000:2.0:1789760792.590625:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.590631:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.590634:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29126656, 29130752) 0 0 00000080:00200000:2.0:1789760792.590638:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.590651:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7111, 7111] 00020000:00200000:2.0:1789760792.590658:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29126656 chunk: [29126656, 29130752) 29130752, 4096 00020000:00200000:2.0:1789760792.590662:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.590677:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7111, 7111] 00000008:00010000:2.0:1789760792.590690:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.590695:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760792.590704:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29126656, 29130752) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.590714:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29126656=0x1bc7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.590758:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.590761:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29130752 00000020:00010000:2.0:1789760792.590768:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.590771:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.590774:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.590777:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.590779:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.590785:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.590790:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.590802:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760792.590873:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 29126656<>29130752 00000100:00100000:3.0:1789760792.591570:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760792.591589:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512774784:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760792.632551:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760792.632561:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.632569:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.632589:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760792.632613:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512774784:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760792.632696:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7111, 7111] 00000080:00200000:0.0:1789760792.632709:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760792.632722:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760792.632734:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29130752 i_blocks 57351 00000020:00200000:0.0:1789760792.632738:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760792.632746:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760792.632773:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29130752, count: 4096 00000080:00200000:0.0:1789760792.632779:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29130752, bytes: 4096 00000080:00200000:0.0:1789760792.632796:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760792.632803:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760792.632805:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29130752, 29134848) 0 0 00000080:00200000:0.0:1789760792.632809:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760792.632822:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7112, 7112] 00020000:00200000:0.0:1789760792.632829:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29130752 chunk: [29130752, 29134848) 29134848, 4096 00020000:00200000:0.0:1789760792.632832:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760792.632847:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7112, 7112] 00000008:00010000:0.0:1789760792.632860:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760792.632864:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760792.632873:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29130752, 29134848) append 0 append lockpos 0 00000080:00200000:0.0:1789760792.632883:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29130752=0x1bc8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760792.632970:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760792.632973:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29134848 00000020:00010000:0.0:1789760792.632980:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760792.632984:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.632987:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760792.632990:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760792.632993:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760792.632999:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760792.633005:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760792.633019:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccb40 00000008:00100000:0.0:1789760792.633110:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514ad80 - object 0x240000400:5 offset 29130752<>29134848 00000100:00100000:0.0:1789760792.633151:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760792.633169:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512774912:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760792.707610:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974430 gen 3 00000100:00100000:0.0:1789760792.707642:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00514ad80 x1876699512774912/t4294974431(4294974431) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760808 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:0.0:1789760792.707679:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:0.0:1789760792.707686:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760792.707694:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760792.707716:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:0.0:1789760792.707735:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512774912:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760792.707814:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7112, 7112] 00000080:00200000:2.0:1789760792.707826:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.707831:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.707843:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29134848 i_blocks 57351 00000020:00200000:2.0:1789760792.707846:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.707853:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.707878:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29134848, count: 4096 00000080:00200000:2.0:1789760792.707885:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29134848, bytes: 4096 00000080:00200000:2.0:1789760792.707893:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.707928:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.707931:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29134848, 29138944) 0 0 00000080:00200000:2.0:1789760792.707934:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.707957:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7113, 7113] 00020000:00200000:2.0:1789760792.707964:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29134848 chunk: [29134848, 29138944) 29138944, 4096 00020000:00200000:2.0:1789760792.707967:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.707982:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7113, 7113] 00000008:00010000:2.0:1789760792.707996:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.708001:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760792.708010:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29134848, 29138944) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.708020:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29134848=0x1bc9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.708074:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.708077:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29138944 00000020:00010000:2.0:1789760792.708085:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.708088:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.708091:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.708094:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.708096:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.708102:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.708108:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.708120:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760792.708191:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 29134848<>29138944 00000100:00100000:3.0:1789760792.708336:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760792.708351:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512775040:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760792.758953:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974432 gen 3 00000008:00010000:3.0:1789760792.758998:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760792.759004:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.759011:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.759029:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760792.759062:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512775040:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760792.759143:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7113, 7113] 00000080:00200000:2.0:1789760792.759155:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.759161:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.759173:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29138944 i_blocks 57351 00000020:00200000:2.0:1789760792.759177:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.759185:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.759211:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29138944, count: 4096 00000080:00200000:2.0:1789760792.759218:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29138944, bytes: 4096 00000080:00200000:2.0:1789760792.759225:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.759232:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.759234:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29138944, 29143040) 0 0 00000080:00200000:2.0:1789760792.759238:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.759251:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7114, 7114] 00020000:00200000:2.0:1789760792.759258:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29138944 chunk: [29138944, 29143040) 29143040, 4096 00020000:00200000:2.0:1789760792.759261:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.759276:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7114, 7114] 00000008:00010000:2.0:1789760792.759289:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.759294:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760792.759303:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29138944, 29143040) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.759313:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29138944=0x1bca000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.759359:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.759362:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29143040 00000020:00010000:2.0:1789760792.759368:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.759371:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.759374:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.759377:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.759380:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.759386:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.759391:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.759404:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760792.759640:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 29138944<>29143040 00000100:00100000:3.0:1789760792.759739:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760792.759753:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512775168:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760792.799724:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974433 gen 3 00000008:00010000:3.0:1789760792.799760:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760792.799766:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.799774:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.799827:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760792.799850:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512775168:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760792.800197:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7114, 7114] 00000080:00200000:2.0:1789760792.800210:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.800215:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.800227:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29143040 i_blocks 57351 00000020:00200000:2.0:1789760792.800230:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.800238:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.800263:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29143040, count: 4096 00000080:00200000:2.0:1789760792.800268:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29143040, bytes: 4096 00000080:00200000:2.0:1789760792.800276:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.800284:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.800286:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29143040, 29147136) 0 0 00000080:00200000:2.0:1789760792.800289:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.800303:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7115, 7115] 00020000:00200000:2.0:1789760792.800309:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29143040 chunk: [29143040, 29147136) 29147136, 4096 00020000:00200000:2.0:1789760792.800313:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.800327:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7115, 7115] 00000008:00010000:2.0:1789760792.800341:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.800346:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760792.800355:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29143040, 29147136) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.800364:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29143040=0x1bcb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.800409:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.800412:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29147136 00000020:00010000:2.0:1789760792.800418:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.800422:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.800425:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.800428:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.800506:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.800513:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.800519:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.800532:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760792.800647:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 29143040<>29147136 00000100:00100000:3.0:1789760792.800697:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512775296:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760792.838988:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974434 gen 3 00000008:00010000:3.0:1789760792.839034:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760792.839050:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.839059:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.839068:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760792.839090:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512775296:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760792.839271:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7115, 7115] 00000080:00200000:2.0:1789760792.839284:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.839288:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.839299:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29147136 i_blocks 57351 00000020:00200000:2.0:1789760792.839302:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.839309:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.839332:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29147136, count: 4096 00000080:00200000:2.0:1789760792.839337:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29147136, bytes: 4096 00000080:00200000:2.0:1789760792.839344:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.839350:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.839352:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29147136, 29151232) 0 0 00000080:00200000:2.0:1789760792.839356:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.839367:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7116, 7116] 00020000:00200000:2.0:1789760792.839374:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29147136 chunk: [29147136, 29151232) 29151232, 4096 00020000:00200000:2.0:1789760792.839377:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.839390:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7116, 7116] 00000008:00010000:2.0:1789760792.839403:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.839407:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760792.839416:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29147136, 29151232) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.839426:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29147136=0x1bcc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.839539:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.839542:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29151232 00000020:00010000:2.0:1789760792.839549:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.839553:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.839556:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.839559:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.839561:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.839568:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.839573:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.839587:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760792.839867:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 29147136<>29151232 00000100:00100000:3.0:1789760792.840326:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512775424:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760792.909265:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760792.909277:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.909284:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.909301:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760792.909320:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512775424:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760792.909452:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7116, 7116] 00000080:00200000:2.0:1789760792.909465:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.909470:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.909481:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29151232 i_blocks 57351 00000020:00200000:2.0:1789760792.909484:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.909492:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.909519:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29151232, count: 4096 00000080:00200000:2.0:1789760792.909525:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29151232, bytes: 4096 00000080:00200000:2.0:1789760792.909532:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.909539:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.909542:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29151232, 29155328) 0 0 00000080:00200000:2.0:1789760792.909545:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.909557:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7117, 7117] 00020000:00200000:2.0:1789760792.909564:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29151232 chunk: [29151232, 29155328) 29155328, 4096 00020000:00200000:2.0:1789760792.909567:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.909581:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7117, 7117] 00000008:00010000:2.0:1789760792.909594:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.909598:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760792.909607:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29151232, 29155328) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.909749:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29151232=0x1bcd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.909795:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.909798:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29155328 00000020:00010000:2.0:1789760792.909805:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.909808:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.909811:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.909814:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.909816:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.909823:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.909828:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.909843:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760792.909955:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 29151232<>29155328 00000100:00100000:3.0:1789760792.910097:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760792.910112:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512775552:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760792.939453:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974436 gen 3 00000008:00010000:3.0:1789760792.939497:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760792.939503:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.939510:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.939526:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760792.939548:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512775552:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760792.939821:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7117, 7117] 00000080:00200000:2.0:1789760792.939834:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.939839:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.939850:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29155328 i_blocks 57351 00000020:00200000:2.0:1789760792.939853:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.939860:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.940032:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29155328, count: 4096 00000080:00200000:2.0:1789760792.940049:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29155328, bytes: 4096 00000080:00200000:2.0:1789760792.940058:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.940065:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.940067:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29155328, 29159424) 0 0 00000080:00200000:2.0:1789760792.940071:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.940084:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7118, 7118] 00020000:00200000:2.0:1789760792.940090:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29155328 chunk: [29155328, 29159424) 29159424, 4096 00020000:00200000:2.0:1789760792.940094:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.940108:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7118, 7118] 00000008:00010000:2.0:1789760792.940121:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.940125:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760792.940133:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29155328, 29159424) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.940142:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29155328=0x1bce000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.940180:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.940184:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29159424 00000020:00010000:2.0:1789760792.940190:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.940193:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.940196:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.940199:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.940201:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.940277:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.940284:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.940299:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760792.940375:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 29155328<>29159424 00000100:00100000:3.0:1789760792.940448:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512775680:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760792.979185:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974437 gen 3 00000008:00010000:3.0:1789760792.979222:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760792.979229:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760792.979237:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760792.979254:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760792.979278:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512775680:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760792.979364:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7118, 7118] 00000080:00200000:2.0:1789760792.979378:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760792.979382:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760792.979394:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29159424 i_blocks 57351 00000020:00200000:2.0:1789760792.979398:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760792.979407:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760792.979433:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29159424, count: 4096 00000080:00200000:2.0:1789760792.979439:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29159424, bytes: 4096 00000080:00200000:2.0:1789760792.979446:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760792.979453:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760792.979455:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29159424, 29163520) 0 0 00000080:00200000:2.0:1789760792.979459:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760792.979472:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7119, 7119] 00020000:00200000:2.0:1789760792.979479:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29159424 chunk: [29159424, 29163520) 29163520, 4096 00020000:00200000:2.0:1789760792.979483:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760792.979497:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7119, 7119] 00000008:00010000:2.0:1789760792.979510:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760792.979515:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760792.979524:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29159424, 29163520) append 0 append lockpos 0 00000080:00200000:2.0:1789760792.979534:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29159424=0x1bcf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760792.979578:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760792.979581:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29163520 00000020:00010000:2.0:1789760792.979589:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760792.979592:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.979595:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760792.979598:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760792.979602:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760792.979608:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760792.979690:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760792.979791:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760792.979871:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 29159424<>29163520 00000100:00100000:3.0:1789760792.979936:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512775808:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760793.018623:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974438 gen 3 00000008:00010000:3.0:1789760793.018661:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760793.018669:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.018677:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.018850:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7119, 7119] 00000008:00010000:3.0:1789760793.018936:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000080:00200000:2.0:1789760793.018959:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:3.0:1789760793.018963:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512775808:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760793.018967:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.018981:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29163520 i_blocks 57351 00000020:00200000:2.0:1789760793.018985:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.018994:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.019022:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29163520, count: 4096 00000080:00200000:2.0:1789760793.019028:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29163520, bytes: 4096 00000080:00200000:2.0:1789760793.019050:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.019058:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.019060:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29163520, 29167616) 0 0 00000080:00200000:2.0:1789760793.019064:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.019079:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7120, 7120] 00020000:00200000:2.0:1789760793.019086:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29163520 chunk: [29163520, 29167616) 29167616, 4096 00020000:00200000:2.0:1789760793.019089:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.019104:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7120, 7120] 00000008:00010000:2.0:1789760793.019117:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.019122:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760793.019131:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29163520, 29167616) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.019143:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29163520=0x1bd0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.019187:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.019190:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29167616 00000020:00010000:2.0:1789760793.019197:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.019200:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.019203:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.019205:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.019208:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.019214:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.019219:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.019232:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760793.019299:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 29163520<>29167616 00000100:00100000:3.0:1789760793.020541:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512775936:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760793.067299:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760793.067310:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.067320:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.067337:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760793.067362:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512775936:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760793.067446:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7120, 7120] 00000080:00200000:2.0:1789760793.067458:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.067463:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.067474:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29167616 i_blocks 57351 00000020:00200000:2.0:1789760793.067477:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.067484:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.067509:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29167616, count: 4096 00000080:00200000:2.0:1789760793.067514:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29167616, bytes: 4096 00000080:00200000:2.0:1789760793.067522:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.067529:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.067531:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29167616, 29171712) 0 0 00000080:00200000:2.0:1789760793.067535:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.067547:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7121, 7121] 00020000:00200000:2.0:1789760793.067554:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29167616 chunk: [29167616, 29171712) 29171712, 4096 00020000:00200000:2.0:1789760793.067558:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.067572:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7121, 7121] 00000008:00010000:2.0:1789760793.067584:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.067588:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760793.067597:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29167616, 29171712) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.067608:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29167616=0x1bd1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.067651:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.067654:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29171712 00000020:00010000:2.0:1789760793.067661:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.067664:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.067667:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.067670:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.067672:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.067679:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.067684:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.067697:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760793.067779:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 29167616<>29171712 00000100:00100000:3.0:1789760793.067969:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760793.067981:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512776064:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760793.106325:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974440 gen 3 00000008:00010000:3.0:1789760793.106370:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760793.106378:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.106386:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.106404:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760793.106439:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512776064:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760793.106626:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7121, 7121] 00000080:00200000:2.0:1789760793.106639:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.106644:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.106656:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29171712 i_blocks 57351 00000020:00200000:2.0:1789760793.106660:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.106668:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.106693:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29171712, count: 4096 00000080:00200000:2.0:1789760793.106699:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29171712, bytes: 4096 00000080:00200000:2.0:1789760793.106707:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.106714:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.106717:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29171712, 29175808) 0 0 00000080:00200000:2.0:1789760793.106720:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.106733:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7122, 7122] 00020000:00200000:2.0:1789760793.106740:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29171712 chunk: [29171712, 29175808) 29175808, 4096 00020000:00200000:2.0:1789760793.106744:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.106759:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7122, 7122] 00000008:00010000:2.0:1789760793.106772:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.106777:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760793.106785:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29171712, 29175808) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.106795:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29171712=0x1bd2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.106838:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.106842:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29175808 00000020:00010000:2.0:1789760793.106849:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.106852:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.106855:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.106858:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.106861:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.106868:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.106875:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.106888:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760793.107014:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 29171712<>29175808 00000100:00100000:3.0:1789760793.107116:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760793.107143:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512776192:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760793.143261:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760793.143273:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.143282:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.143301:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760793.143326:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512776192:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760793.143408:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7122, 7122] 00000080:00200000:2.0:1789760793.143420:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.143426:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.143448:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29175808 i_blocks 57351 00000020:00200000:2.0:1789760793.143452:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.143460:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.143484:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29175808, count: 4096 00000080:00200000:2.0:1789760793.143489:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29175808, bytes: 4096 00000080:00200000:2.0:1789760793.143497:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.143504:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.143506:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29175808, 29179904) 0 0 00000080:00200000:2.0:1789760793.143510:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.143524:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7123, 7123] 00020000:00200000:2.0:1789760793.143530:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29175808 chunk: [29175808, 29179904) 29179904, 4096 00020000:00200000:2.0:1789760793.143534:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.143548:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7123, 7123] 00000008:00010000:2.0:1789760793.143561:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.143565:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760793.143574:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29175808, 29179904) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.143584:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29175808=0x1bd3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.143626:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.143630:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29179904 00000020:00010000:2.0:1789760793.143636:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.143640:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.143643:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.143646:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.143648:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.143654:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.143660:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.143673:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760793.143865:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 29175808<>29179904 00000100:00100000:3.0:1789760793.143971:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760793.143984:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512776320:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760793.174553:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974441 gen 3 00000100:00100000:3.0:1789760793.174582:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368d880 x1876699512776320/t4294974442(4294974442) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760809 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760793.174618:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760793.174624:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.174632:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.174648:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760793.174667:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512776320:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760793.174802:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7123, 7123] 00000080:00200000:2.0:1789760793.174814:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.174819:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.174830:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29179904 i_blocks 57351 00000020:00200000:2.0:1789760793.174833:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.174840:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.174864:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29179904, count: 4096 00000080:00200000:2.0:1789760793.174869:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29179904, bytes: 4096 00000080:00200000:2.0:1789760793.174877:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.174884:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.174885:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29179904, 29184000) 0 0 00000080:00200000:2.0:1789760793.174889:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.174924:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7124, 7124] 00020000:00200000:2.0:1789760793.174931:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29179904 chunk: [29179904, 29184000) 29184000, 4096 00020000:00200000:2.0:1789760793.174936:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.174952:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7124, 7124] 00000008:00010000:2.0:1789760793.174967:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.174972:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760793.174982:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29179904, 29184000) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.174993:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29179904=0x1bd4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.175035:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.175050:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29184000 00000020:00010000:2.0:1789760793.175057:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.175060:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.175063:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.175066:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.175069:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.175075:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.175081:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.175094:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760793.175181:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 29179904<>29184000 00000100:00100000:3.0:1789760793.175329:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760793.175343:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512776448:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760793.207113:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974442 gen 3 00000100:00100000:3.0:1789760793.207154:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368f100 x1876699512776448/t4294974443(4294974443) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760809 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760793.207192:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760793.207198:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.207205:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.207224:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760793.207245:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512776448:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760793.207391:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7124, 7124] 00000080:00200000:2.0:1789760793.207405:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.207410:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.207422:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29184000 i_blocks 57351 00000020:00200000:2.0:1789760793.207425:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.207502:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.207528:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29184000, count: 4096 00000080:00200000:2.0:1789760793.207534:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29184000, bytes: 4096 00000080:00200000:2.0:1789760793.207542:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.207548:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.207550:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29184000, 29188096) 0 0 00000080:00200000:2.0:1789760793.207554:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.207567:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7125, 7125] 00020000:00200000:2.0:1789760793.207574:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29184000 chunk: [29184000, 29188096) 29188096, 4096 00020000:00200000:2.0:1789760793.207577:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.207592:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7125, 7125] 00000008:00010000:2.0:1789760793.207605:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.207610:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760793.207620:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29184000, 29188096) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.207630:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29184000=0x1bd5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.207674:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.207677:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29188096 00000020:00010000:2.0:1789760793.207684:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.207687:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.207690:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.207693:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.207696:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.207703:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.207718:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.207733:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760793.207827:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 29184000<>29188096 00000100:00100000:3.0:1789760793.207981:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760793.207996:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512776576:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760793.239506:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974444 gen 3 00000008:00010000:3.0:1789760793.239557:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760793.239563:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.239571:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.239744:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760793.239769:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512776576:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760793.240071:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7125, 7125] 00000080:00200000:2.0:1789760793.240085:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.240090:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.240101:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29188096 i_blocks 57351 00000020:00200000:2.0:1789760793.240104:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.240113:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.240140:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29188096, count: 4096 00000080:00200000:2.0:1789760793.240146:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29188096, bytes: 4096 00000080:00200000:2.0:1789760793.240154:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.240160:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.240162:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29188096, 29192192) 0 0 00000080:00200000:2.0:1789760793.240166:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.240179:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7126, 7126] 00020000:00200000:2.0:1789760793.240185:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29188096 chunk: [29188096, 29192192) 29192192, 4096 00020000:00200000:2.0:1789760793.240189:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.240205:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7126, 7126] 00000008:00010000:2.0:1789760793.240218:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.240223:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760793.240231:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29188096, 29192192) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.240242:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29188096=0x1bd6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.240286:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.240289:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29192192 00000020:00010000:2.0:1789760793.240295:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.240299:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.240302:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.240305:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.240308:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.240314:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.240320:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.240334:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760793.240410:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 29188096<>29192192 00000100:00100000:3.0:1789760793.240504:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512776704:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760793.266490:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974445 gen 3 00000008:00010000:3.0:1789760793.266528:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760793.266535:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.266543:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.266598:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760793.266623:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512776704:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760793.266700:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7126, 7126] 00000080:00200000:2.0:1789760793.266713:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.266718:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.266730:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29192192 i_blocks 57351 00000020:00200000:2.0:1789760793.266734:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.266742:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.266769:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29192192, count: 4096 00000080:00200000:2.0:1789760793.266780:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29192192, bytes: 4096 00000080:00200000:2.0:1789760793.266833:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.266841:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.266843:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29192192, 29196288) 0 0 00000080:00200000:2.0:1789760793.266847:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.266955:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7127, 7127] 00020000:00200000:2.0:1789760793.266962:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29192192 chunk: [29192192, 29196288) 29196288, 4096 00020000:00200000:2.0:1789760793.266966:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.266981:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7127, 7127] 00000008:00010000:2.0:1789760793.266994:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.266998:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760793.267006:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29192192, 29196288) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.267016:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29192192=0x1bd7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.267065:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.267068:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29196288 00000020:00010000:2.0:1789760793.267075:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.267078:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.267081:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.267084:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.267087:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.267093:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.267098:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.267112:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760793.267189:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 29192192<>29196288 00000100:00100000:3.0:1789760793.267243:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512776832:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760793.303218:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974446 gen 3 00000008:00010000:3.0:1789760793.303252:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760793.303259:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.303267:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.303288:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760793.303313:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512776832:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760793.303405:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7127, 7127] 00000080:00200000:2.0:1789760793.303417:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.303422:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.303441:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29196288 i_blocks 57351 00000020:00200000:2.0:1789760793.303445:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.303453:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.303478:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29196288, count: 4096 00000080:00200000:2.0:1789760793.303483:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29196288, bytes: 4096 00000080:00200000:2.0:1789760793.303491:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.303498:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.303500:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29196288, 29200384) 0 0 00000080:00200000:2.0:1789760793.303504:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.303516:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7128, 7128] 00020000:00200000:2.0:1789760793.303523:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29196288 chunk: [29196288, 29200384) 29200384, 4096 00020000:00200000:2.0:1789760793.303527:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.303543:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7128, 7128] 00000008:00010000:2.0:1789760793.303555:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.303560:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760793.303568:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29196288, 29200384) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.303578:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29196288=0x1bd8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.303620:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.303624:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29200384 00000020:00010000:2.0:1789760793.303631:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.303634:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.303637:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.303640:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.303642:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.303648:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.303654:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.303666:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760793.303749:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 29196288<>29200384 00000100:00100000:3.0:1789760793.303926:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512776960:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760793.334793:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760793.334804:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.334814:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.334834:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760793.334856:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512776960:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760793.334948:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7128, 7128] 00000080:00200000:2.0:1789760793.334961:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.334966:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.334977:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29200384 i_blocks 57351 00000020:00200000:2.0:1789760793.334981:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.334987:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.335008:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29200384, count: 4096 00000080:00200000:2.0:1789760793.335013:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29200384, bytes: 4096 00000080:00200000:2.0:1789760793.335019:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.335024:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.335026:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29200384, 29204480) 0 0 00000080:00200000:2.0:1789760793.335030:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.335051:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7129, 7129] 00020000:00200000:2.0:1789760793.335057:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29200384 chunk: [29200384, 29204480) 29204480, 4096 00020000:00200000:2.0:1789760793.335060:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.335075:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7129, 7129] 00000008:00010000:2.0:1789760793.335089:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.335094:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760793.335102:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29200384, 29204480) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.335113:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29200384=0x1bd9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.335154:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.335157:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29204480 00000020:00010000:2.0:1789760793.335163:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.335166:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.335168:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.335171:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.335173:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.335178:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.335183:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.335196:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760793.335272:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 29200384<>29204480 00000100:00100000:3.0:1789760793.335367:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512777088:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760793.362761:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974447 gen 3 00000100:00100000:3.0:1789760793.362802:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368ca80 x1876699512777088/t4294974448(4294974448) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760809 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760793.362841:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760793.362847:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.362854:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.362873:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760793.362893:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512777088:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760793.363111:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7129, 7129] 00000080:00200000:2.0:1789760793.363124:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.363129:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.363140:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29204480 i_blocks 57351 00000020:00200000:2.0:1789760793.363143:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.363151:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.363176:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29204480, count: 4096 00000080:00200000:2.0:1789760793.363183:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29204480, bytes: 4096 00000080:00200000:2.0:1789760793.363191:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.363198:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.363200:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29204480, 29208576) 0 0 00000080:00200000:2.0:1789760793.363203:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.363217:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7130, 7130] 00020000:00200000:2.0:1789760793.363224:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29204480 chunk: [29204480, 29208576) 29208576, 4096 00020000:00200000:2.0:1789760793.363227:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.363242:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7130, 7130] 00000008:00010000:2.0:1789760793.363255:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.363260:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760793.363268:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29204480, 29208576) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.363280:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29204480=0x1bda000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.363327:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.363329:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29208576 00000020:00010000:2.0:1789760793.363335:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.363337:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.363339:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.363341:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.363343:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.363347:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.363407:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.363475:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760793.363637:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 29204480<>29208576 00000100:00100000:3.0:1789760793.363692:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512777216:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760793.373524:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760793.415978:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974449 gen 3 00000008:00010000:3.0:1789760793.416026:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760793.416032:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.416052:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.416070:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760793.416092:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512777216:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760793.416357:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7130, 7130] 00000080:00200000:2.0:1789760793.416370:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.416375:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.416387:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29208576 i_blocks 57351 00000020:00200000:2.0:1789760793.416390:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.416398:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.416422:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29208576, count: 4096 00000080:00200000:2.0:1789760793.416427:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29208576, bytes: 4096 00000080:00200000:2.0:1789760793.416435:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.416442:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.416444:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29208576, 29212672) 0 0 00000080:00200000:2.0:1789760793.416447:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.416461:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7131, 7131] 00020000:00200000:2.0:1789760793.416468:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29208576 chunk: [29208576, 29212672) 29212672, 4096 00020000:00200000:2.0:1789760793.416472:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.416487:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7131, 7131] 00000008:00010000:2.0:1789760793.416500:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.416504:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760793.416513:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29208576, 29212672) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.416522:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29208576=0x1bdb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.416569:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.416572:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29212672 00000020:00010000:2.0:1789760793.416579:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.416582:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.416585:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.416588:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.416591:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.416597:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.416602:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.416615:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760793.416760:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 29208576<>29212672 00000100:00100000:3.0:1789760793.416852:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512777344:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760793.476170:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974450 gen 3 00000008:00010000:3.0:1789760793.476209:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760793.476216:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.476226:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.476245:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760793.476269:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512777344:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760793.476440:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7131, 7131] 00000080:00200000:2.0:1789760793.476454:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.476458:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.476470:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29212672 i_blocks 57351 00000020:00200000:2.0:1789760793.476474:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.476483:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.476508:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29212672, count: 4096 00000080:00200000:2.0:1789760793.476515:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29212672, bytes: 4096 00000080:00200000:2.0:1789760793.476524:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.476531:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.476533:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29212672, 29216768) 0 0 00000080:00200000:2.0:1789760793.476655:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.476670:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7132, 7132] 00020000:00200000:2.0:1789760793.476678:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29212672 chunk: [29212672, 29216768) 29216768, 4096 00020000:00200000:2.0:1789760793.476681:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.476698:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7132, 7132] 00000008:00010000:2.0:1789760793.476719:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.476724:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760793.476733:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29212672, 29216768) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.476744:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29212672=0x1bdc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.477018:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.477021:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29216768 00000020:00010000:2.0:1789760793.477029:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.477032:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.477044:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.477048:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.477051:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.477057:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.477063:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.477079:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760793.477161:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 29212672<>29216768 00000100:00100000:3.0:1789760793.477309:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512777472:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760793.520982:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974451 gen 3 00000008:00010000:3.0:1789760793.521020:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760793.521027:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.521044:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.521105:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760793.521130:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512777472:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760793.521323:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7132, 7132] 00000080:00200000:2.0:1789760793.521336:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.521341:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.521353:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29216768 i_blocks 57351 00000020:00200000:2.0:1789760793.521357:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.521364:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.521386:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29216768, count: 4096 00000080:00200000:2.0:1789760793.521390:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29216768, bytes: 4096 00000080:00200000:2.0:1789760793.521396:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.521401:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.521403:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29216768, 29220864) 0 0 00000080:00200000:2.0:1789760793.521406:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.521416:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7133, 7133] 00020000:00200000:2.0:1789760793.521420:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29216768 chunk: [29216768, 29220864) 29220864, 4096 00020000:00200000:2.0:1789760793.521423:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.521485:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7133, 7133] 00000008:00010000:2.0:1789760793.521496:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.521500:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760793.521507:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29216768, 29220864) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.521516:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29216768=0x1bdd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.521551:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.521552:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29220864 00000020:00010000:2.0:1789760793.521557:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.521560:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.521562:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.521566:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.521568:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.521575:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.521580:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.521593:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760793.521666:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 29216768<>29220864 00000100:00100000:3.0:1789760793.521734:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512777600:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760793.555513:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974452 gen 3 00000008:00010000:3.0:1789760793.555546:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760793.555552:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.555560:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.555612:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760793.555634:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512777600:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760793.555703:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7133, 7133] 00000080:00200000:2.0:1789760793.555769:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.555774:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.555785:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29220864 i_blocks 57351 00000020:00200000:2.0:1789760793.555833:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.555841:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.555866:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29220864, count: 4096 00000080:00200000:2.0:1789760793.555872:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29220864, bytes: 4096 00000080:00200000:2.0:1789760793.555879:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.555886:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.555889:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29220864, 29224960) 0 0 00000080:00200000:2.0:1789760793.555892:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.555929:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7134, 7134] 00020000:00200000:2.0:1789760793.555936:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29220864 chunk: [29220864, 29224960) 29224960, 4096 00020000:00200000:2.0:1789760793.555941:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.556014:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7134, 7134] 00000008:00010000:2.0:1789760793.556029:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.556041:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760793.556051:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29220864, 29224960) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.556061:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29220864=0x1bde000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.556100:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.556102:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29224960 00000020:00010000:2.0:1789760793.556108:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.556112:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.556115:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.556118:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.556121:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.556126:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.556132:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.556143:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760793.556210:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 29220864<>29224960 00000100:00100000:3.0:1789760793.556266:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512777728:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760793.602050:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974453 gen 3 00000008:00010000:2.0:1789760793.602160:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:2.0:1789760793.602167:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760793.602174:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760793.602380:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:2.0:1789760793.602408:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512777728:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760793.602556:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7134, 7134] 00000080:00200000:0.0:1789760793.602569:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760793.602574:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760793.602586:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29224960 i_blocks 57351 00000020:00200000:0.0:1789760793.602590:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760793.602598:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760793.602625:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29224960, count: 4096 00000080:00200000:0.0:1789760793.602631:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29224960, bytes: 4096 00000080:00200000:0.0:1789760793.602639:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760793.602647:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760793.602650:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29224960, 29229056) 0 0 00000080:00200000:0.0:1789760793.603000:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760793.603015:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7135, 7135] 00020000:00200000:0.0:1789760793.603022:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29224960 chunk: [29224960, 29229056) 29229056, 4096 00020000:00200000:0.0:1789760793.603025:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760793.603051:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7135, 7135] 00000008:00010000:0.0:1789760793.603065:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760793.603070:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760793.603080:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29224960, 29229056) append 0 append lockpos 0 00000080:00200000:0.0:1789760793.603091:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29224960=0x1bdf000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760793.603138:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760793.603141:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29229056 00000020:00010000:0.0:1789760793.603147:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760793.603151:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.603154:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.603157:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760793.603160:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760793.603166:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760793.603171:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760793.603187:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccb40 00000008:00100000:0.0:1789760793.603940:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148e00 - object 0x240000400:5 offset 29224960<>29229056 00000100:00100000:1.0:1789760793.604000:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512777856:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760793.644515:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974454 gen 3 00000008:00010000:1.0:1789760793.644555:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:1.0:1789760793.644561:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760793.644729:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760793.644799:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:1.0:1789760793.644823:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512777856:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760793.645005:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7135, 7135] 00000080:00200000:0.0:1789760793.645018:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760793.645023:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760793.645051:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29229056 i_blocks 57351 00000020:00200000:0.0:1789760793.645055:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760793.645062:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760793.645089:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29229056, count: 4096 00000080:00200000:0.0:1789760793.645095:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29229056, bytes: 4096 00000080:00200000:0.0:1789760793.645102:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760793.645110:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760793.645111:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29229056, 29233152) 0 0 00000080:00200000:0.0:1789760793.645116:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760793.645129:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7136, 7136] 00020000:00200000:0.0:1789760793.645252:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29229056 chunk: [29229056, 29233152) 29233152, 4096 00020000:00200000:0.0:1789760793.645257:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760793.645274:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7136, 7136] 00000008:00010000:0.0:1789760793.645287:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760793.645291:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760793.645300:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29229056, 29233152) append 0 append lockpos 0 00000080:00200000:0.0:1789760793.645310:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29229056=0x1be0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760793.645353:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760793.645355:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29233152 00000020:00010000:0.0:1789760793.645361:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760793.645364:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.645366:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.645369:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760793.645372:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760793.645378:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760793.645384:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760793.645399:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000008:00100000:0.0:1789760793.645475:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a680 - object 0x240000400:5 offset 29229056<>29233152 00000100:00100000:1.0:1789760793.645615:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512777984:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760793.685252:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974455 gen 3 00000008:00010000:1.0:1789760793.685288:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760793.685294:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760793.685302:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760793.685328:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760793.685352:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512777984:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760793.685515:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7136, 7136] 00000080:00200000:0.0:1789760793.685527:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760793.685532:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760793.685544:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29233152 i_blocks 57351 00000020:00200000:0.0:1789760793.685547:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760793.685555:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760793.685580:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29233152, count: 4096 00000080:00200000:0.0:1789760793.685586:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29233152, bytes: 4096 00000080:00200000:0.0:1789760793.685593:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760793.685601:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760793.685603:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29233152, 29237248) 0 0 00000080:00200000:0.0:1789760793.685606:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760793.685620:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7137, 7137] 00020000:00200000:0.0:1789760793.685627:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29233152 chunk: [29233152, 29237248) 29237248, 4096 00020000:00200000:0.0:1789760793.685631:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760793.685646:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7137, 7137] 00000008:00010000:0.0:1789760793.685659:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760793.685663:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760793.685673:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29233152, 29237248) append 0 append lockpos 0 00000080:00200000:0.0:1789760793.685684:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29233152=0x1be1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760793.685867:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760793.685870:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29237248 00000020:00010000:0.0:1789760793.685877:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760793.685881:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.685883:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.685886:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760793.685889:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760793.685895:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760793.686140:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760793.686156:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc900 00000008:00100000:0.0:1789760793.686227:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a300 - object 0x240000400:5 offset 29233152<>29237248 00000100:00100000:1.0:1789760793.686282:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512778112:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760793.718679:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760793.718690:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760793.718700:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760793.718836:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760793.718860:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512778112:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760793.719000:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7137, 7137] 00000080:00200000:0.0:1789760793.719014:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760793.719018:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760793.719130:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29237248 i_blocks 57351 00000020:00200000:0.0:1789760793.719135:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760793.719143:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760793.719171:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29237248, count: 4096 00000080:00200000:0.0:1789760793.719178:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29237248, bytes: 4096 00000080:00200000:0.0:1789760793.719185:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760793.719193:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760793.719195:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29237248, 29241344) 0 0 00000080:00200000:0.0:1789760793.719199:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760793.719212:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7138, 7138] 00020000:00200000:0.0:1789760793.719218:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29237248 chunk: [29237248, 29241344) 29241344, 4096 00020000:00200000:0.0:1789760793.719222:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760793.719237:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7138, 7138] 00000008:00010000:0.0:1789760793.719250:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760793.719255:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760793.719264:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29237248, 29241344) append 0 append lockpos 0 00000080:00200000:0.0:1789760793.719273:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29237248=0x1be2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760793.719313:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760793.719315:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29241344 00000020:00010000:0.0:1789760793.719322:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760793.719325:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.719327:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.719331:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760793.719333:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760793.719339:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760793.719345:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760793.719429:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cccc0 00000008:00100000:0.0:1789760793.719506:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149880 - object 0x240000400:5 offset 29237248<>29241344 00000100:00100000:1.0:1789760793.719860:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760793.719876:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512778240:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760793.762137:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974457 gen 3 00000008:00010000:1.0:1789760793.762184:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760793.762191:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760793.762199:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760793.762213:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760793.762237:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512778240:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760793.762357:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7138, 7138] 00000080:00200000:0.0:1789760793.762368:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760793.762373:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760793.762385:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29241344 i_blocks 57351 00000020:00200000:0.0:1789760793.762389:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760793.762397:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760793.762422:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29241344, count: 4096 00000080:00200000:0.0:1789760793.762440:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29241344, bytes: 4096 00000080:00200000:0.0:1789760793.762448:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760793.762457:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760793.762459:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29241344, 29245440) 0 0 00000080:00200000:0.0:1789760793.762463:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760793.762475:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7139, 7139] 00020000:00200000:0.0:1789760793.762483:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29241344 chunk: [29241344, 29245440) 29245440, 4096 00020000:00200000:0.0:1789760793.762486:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760793.762501:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7139, 7139] 00000008:00010000:0.0:1789760793.762514:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760793.762519:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760793.762527:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29241344, 29245440) append 0 append lockpos 0 00000080:00200000:0.0:1789760793.762538:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29241344=0x1be3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760793.762581:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760793.762584:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29245440 00000020:00010000:0.0:1789760793.762591:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760793.762594:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.762596:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.762599:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760793.762602:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760793.762608:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760793.762613:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760793.762626:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000008:00100000:0.0:1789760793.762873:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148380 - object 0x240000400:5 offset 29241344<>29245440 00000100:00100000:1.0:1789760793.762931:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512778368:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760793.800971:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974458 gen 3 00000008:00010000:1.0:1789760793.801004:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760793.801010:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760793.801018:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760793.801131:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760793.801154:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512778368:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760793.801387:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7139, 7139] 00000080:00200000:0.0:1789760793.801399:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760793.801403:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760793.801414:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29245440 i_blocks 57351 00000020:00200000:0.0:1789760793.801418:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760793.801489:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760793.801516:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29245440, count: 4096 00000080:00200000:0.0:1789760793.801522:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29245440, bytes: 4096 00000080:00200000:0.0:1789760793.801530:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760793.801537:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760793.801539:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29245440, 29249536) 0 0 00000080:00200000:0.0:1789760793.801542:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760793.801555:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7140, 7140] 00020000:00200000:0.0:1789760793.801561:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29245440 chunk: [29245440, 29249536) 29249536, 4096 00020000:00200000:0.0:1789760793.801565:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760793.801578:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7140, 7140] 00000008:00010000:0.0:1789760793.801591:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760793.801595:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760793.801604:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29245440, 29249536) append 0 append lockpos 0 00000080:00200000:0.0:1789760793.801614:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29245440=0x1be4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760793.801654:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760793.801657:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29249536 00000020:00010000:0.0:1789760793.801663:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760793.801666:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.801669:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.801672:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760793.801675:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760793.801681:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760793.801686:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760793.801698:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc600 00000008:00100000:0.0:1789760793.801780:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b800 - object 0x240000400:5 offset 29245440<>29249536 00000100:00100000:1.0:1789760793.801936:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760793.801950:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512778496:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760793.840545:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974459 gen 3 00000008:00010000:1.0:1789760793.840585:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760793.840593:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760793.840603:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760793.840624:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760793.840647:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512778496:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760793.840748:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7140, 7140] 00000080:00200000:0.0:1789760793.840761:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760793.840766:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760793.840777:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29249536 i_blocks 57351 00000020:00200000:0.0:1789760793.840780:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760793.840787:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760793.840813:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29249536, count: 4096 00000080:00200000:0.0:1789760793.840820:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29249536, bytes: 4096 00000080:00200000:0.0:1789760793.840827:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760793.840835:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760793.840837:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29249536, 29253632) 0 0 00000080:00200000:0.0:1789760793.840841:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760793.840855:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7141, 7141] 00020000:00200000:0.0:1789760793.840862:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29249536 chunk: [29249536, 29253632) 29253632, 4096 00020000:00200000:0.0:1789760793.840866:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760793.840881:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7141, 7141] 00000008:00010000:0.0:1789760793.840895:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760793.840923:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760793.840933:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29249536, 29253632) append 0 append lockpos 0 00000080:00200000:0.0:1789760793.840945:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29249536=0x1be5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760793.840990:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760793.840993:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29253632 00000020:00010000:0.0:1789760793.841000:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760793.841003:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.841006:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760793.841009:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760793.841012:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760793.841018:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760793.841023:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760793.841047:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc540 00000008:00100000:0.0:1789760793.841126:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b480 - object 0x240000400:5 offset 29249536<>29253632 00000100:00100000:0.0:1789760793.841165:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1789760793.841178:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512778624:192.168.203.156@tcp:4:dd.0 00000008:00010000:0.0:1789760793.884076:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:0.0:1789760793.884093:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760793.884104:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760793.884124:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:0.0:1789760793.884148:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512778624:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760793.884239:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7141, 7141] 00000080:00200000:2.0:1789760793.884252:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.884257:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.884270:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29253632 i_blocks 57351 00000020:00200000:2.0:1789760793.884273:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.884280:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.884307:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29253632, count: 4096 00000080:00200000:2.0:1789760793.884313:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29253632, bytes: 4096 00000080:00200000:2.0:1789760793.884320:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.884326:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.884328:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29253632, 29257728) 0 0 00000080:00200000:2.0:1789760793.884332:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.884345:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7142, 7142] 00020000:00200000:2.0:1789760793.884351:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29253632 chunk: [29253632, 29257728) 29257728, 4096 00020000:00200000:2.0:1789760793.884354:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.884368:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7142, 7142] 00000008:00010000:2.0:1789760793.884380:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.884385:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760793.884393:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29253632, 29257728) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.884403:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29253632=0x1be6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.884458:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.884461:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29257728 00000020:00010000:2.0:1789760793.884468:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.884471:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.884474:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.884477:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.884479:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.884486:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.884490:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.884503:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760793.884574:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 29253632<>29257728 00000100:00100000:2.0:1789760793.884613:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760793.884624:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512778752:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760793.930679:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974460 gen 3 00000100:00100000:3.0:1789760793.930710:0:2360:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368ed80 x1876699512778752/t4294974461(4294974461) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760809 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760793.930753:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760793.930759:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.930766:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.930833:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760793.930855:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512778752:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760793.930944:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7142, 7142] 00000080:00200000:2.0:1789760793.930957:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.930962:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.930974:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29257728 i_blocks 57351 00000020:00200000:2.0:1789760793.930977:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.930985:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.931009:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29257728, count: 4096 00000080:00200000:2.0:1789760793.931015:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29257728, bytes: 4096 00000080:00200000:2.0:1789760793.931023:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.931030:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.931042:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29257728, 29261824) 0 0 00000080:00200000:2.0:1789760793.931046:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.931059:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7143, 7143] 00020000:00200000:2.0:1789760793.931065:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29257728 chunk: [29257728, 29261824) 29261824, 4096 00020000:00200000:2.0:1789760793.931069:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.931084:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7143, 7143] 00000008:00010000:2.0:1789760793.931098:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.931103:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760793.931113:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29257728, 29261824) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.931216:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29257728=0x1be7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.931261:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.931264:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29261824 00000020:00010000:2.0:1789760793.931271:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.931274:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.931277:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.931280:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.931283:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.931289:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.931294:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.931358:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760793.931477:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 29257728<>29261824 00000100:00100000:3.0:1789760793.931519:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512778880:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760793.964551:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974462 gen 3 00000008:00010000:3.0:1789760793.964607:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760793.964615:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760793.964624:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760793.964686:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760793.964715:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512778880:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760793.968534:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7143, 7143] 00000080:00200000:2.0:1789760793.968563:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760793.968570:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760793.968589:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29261824 i_blocks 57351 00000020:00200000:2.0:1789760793.968594:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760793.968611:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760793.968646:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29261824, count: 4096 00000080:00200000:2.0:1789760793.968654:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29261824, bytes: 4096 00000080:00200000:2.0:1789760793.968669:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760793.968678:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760793.968680:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29261824, 29265920) 0 0 00000080:00200000:2.0:1789760793.968685:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760793.969316:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7144, 7144] 00020000:00200000:2.0:1789760793.969384:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29261824 chunk: [29261824, 29265920) 29265920, 4096 00020000:00200000:2.0:1789760793.969390:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760793.969415:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7144, 7144] 00000008:00010000:2.0:1789760793.969443:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760793.969449:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760793.969461:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29261824, 29265920) append 0 append lockpos 0 00000080:00200000:2.0:1789760793.969476:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29261824=0x1be8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760793.969542:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760793.969544:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29265920 00000020:00010000:2.0:1789760793.969552:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760793.969556:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.969558:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760793.969561:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760793.969564:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760793.969573:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760793.969579:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760793.969608:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760793.969838:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 29261824<>29265920 00000100:00100000:3.0:1789760793.970103:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512779008:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760794.025335:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974463 gen 3 00000008:00010000:3.0:1789760794.025390:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760794.025408:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.025418:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.025576:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760794.025613:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512779008:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760794.026006:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7144, 7144] 00000080:00200000:2.0:1789760794.026023:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760794.026029:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760794.026055:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29265920 i_blocks 57351 00000020:00200000:2.0:1789760794.026059:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760794.026125:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760794.026162:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29265920, count: 4096 00000080:00200000:2.0:1789760794.026170:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29265920, bytes: 4096 00000080:00200000:2.0:1789760794.026180:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760794.026190:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760794.026193:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29265920, 29270016) 0 0 00000080:00200000:2.0:1789760794.026198:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760794.026215:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7145, 7145] 00020000:00200000:2.0:1789760794.026223:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29265920 chunk: [29265920, 29270016) 29270016, 4096 00020000:00200000:2.0:1789760794.026228:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760794.026249:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7145, 7145] 00000008:00010000:2.0:1789760794.026268:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760794.026274:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760794.026286:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29265920, 29270016) append 0 append lockpos 0 00000080:00200000:2.0:1789760794.026303:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29265920=0x1be9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760794.028676:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760794.028689:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29270016 00000020:00010000:2.0:1789760794.028705:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760794.028710:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.028714:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760794.028718:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760794.028721:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760794.028734:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760794.028744:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760794.028898:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760794.029056:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 29265920<>29270016 00000100:00100000:2.0:1789760794.029303:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760794.029342:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512779136:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760794.097185:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760794.097200:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.097210:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.097806:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760794.097846:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512779136:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760794.098014:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7145, 7145] 00000080:00200000:0.0:1789760794.098039:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.098045:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.098200:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29270016 i_blocks 57351 00000020:00200000:0.0:1789760794.098205:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.098216:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.098252:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29270016, count: 4096 00000080:00200000:0.0:1789760794.098260:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29270016, bytes: 4096 00000080:00200000:0.0:1789760794.098275:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.098284:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.098286:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29270016, 29274112) 0 0 00000080:00200000:0.0:1789760794.098291:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.098308:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7146, 7146] 00020000:00200000:0.0:1789760794.098316:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29270016 chunk: [29270016, 29274112) 29274112, 4096 00020000:00200000:0.0:1789760794.098332:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.098353:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7146, 7146] 00000008:00010000:0.0:1789760794.098370:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.098376:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760794.098386:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29270016, 29274112) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.098409:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29270016=0x1bea000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.098465:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.098469:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29274112 00000020:00010000:0.0:1789760794.098476:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.098480:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.098483:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.098487:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.098490:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.098497:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.098504:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.098518:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc540 00000008:00100000:0.0:1789760794.098831:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149180 - object 0x240000400:5 offset 29270016<>29274112 00000100:00100000:1.0:1789760794.098951:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760794.098979:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512779264:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760794.138579:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974465 gen 3 00000008:00010000:1.0:1789760794.138626:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760794.138632:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.138640:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.138665:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760794.138689:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512779264:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760794.138849:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7146, 7146] 00000080:00200000:0.0:1789760794.138862:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.138866:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.138878:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29274112 i_blocks 57351 00000020:00200000:0.0:1789760794.138881:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.138889:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.138949:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29274112, count: 4096 00000080:00200000:0.0:1789760794.138956:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29274112, bytes: 4096 00000080:00200000:0.0:1789760794.138963:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.138969:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.138971:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29274112, 29278208) 0 0 00000080:00200000:0.0:1789760794.138975:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.138987:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7147, 7147] 00020000:00200000:0.0:1789760794.138992:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29274112 chunk: [29274112, 29278208) 29278208, 4096 00020000:00200000:0.0:1789760794.138996:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.139010:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7147, 7147] 00000008:00010000:0.0:1789760794.139187:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.139192:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760794.139203:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29274112, 29278208) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.139214:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29274112=0x1beb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.139258:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.139261:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29278208 00000020:00010000:0.0:1789760794.139267:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.139271:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.139273:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.139276:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.139279:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.139285:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.139290:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.139305:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cca80 00000008:00100000:0.0:1789760794.139378:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148000 - object 0x240000400:5 offset 29274112<>29278208 00000100:00100000:1.0:1789760794.139471:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512779392:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760794.194980:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974466 gen 3 00000008:00010000:1.0:1789760794.195121:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760794.195129:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.195139:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.195158:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760794.195181:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512779392:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760794.195298:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7147, 7147] 00000080:00200000:0.0:1789760794.195312:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.195316:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.195330:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29278208 i_blocks 57351 00000020:00200000:0.0:1789760794.195334:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.195342:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.195372:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29278208, count: 4096 00000080:00200000:0.0:1789760794.195378:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29278208, bytes: 4096 00000080:00200000:0.0:1789760794.195386:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.195393:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.195395:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29278208, 29282304) 0 0 00000080:00200000:0.0:1789760794.195399:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.195412:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7148, 7148] 00020000:00200000:0.0:1789760794.195418:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29278208 chunk: [29278208, 29282304) 29282304, 4096 00020000:00200000:0.0:1789760794.195599:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.195617:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7148, 7148] 00000008:00010000:0.0:1789760794.195632:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.195638:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760794.195649:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29278208, 29282304) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.195663:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29278208=0x1bec000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.195710:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.195713:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29282304 00000020:00010000:0.0:1789760794.195719:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.195722:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.195725:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.195728:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.195730:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.195736:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.195742:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.195756:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccf00 00000008:00100000:0.0:1789760794.195838:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149c00 - object 0x240000400:5 offset 29278208<>29282304 00000100:00100000:1.0:1789760794.196189:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512779520:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760794.242136:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974467 gen 3 00000008:00010000:1.0:1789760794.242173:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760794.242179:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.242189:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.242212:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760794.242236:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512779520:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760794.242338:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7148, 7148] 00000080:00200000:0.0:1789760794.242353:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.242358:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.242370:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29282304 i_blocks 57351 00000020:00200000:0.0:1789760794.242373:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.242381:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.242406:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29282304, count: 4096 00000080:00200000:0.0:1789760794.242413:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29282304, bytes: 4096 00000080:00200000:0.0:1789760794.242421:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.242429:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.242431:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29282304, 29286400) 0 0 00000080:00200000:0.0:1789760794.242434:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.242447:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7149, 7149] 00020000:00200000:0.0:1789760794.242454:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29282304 chunk: [29282304, 29286400) 29286400, 4096 00020000:00200000:0.0:1789760794.242458:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.242473:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7149, 7149] 00000008:00010000:0.0:1789760794.242489:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.242494:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760794.242504:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29282304, 29286400) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.242515:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29282304=0x1bed000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.242559:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.242561:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29286400 00000020:00010000:0.0:1789760794.242568:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.242571:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.242574:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.242577:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.242580:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.242586:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.242592:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.242606:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc000 00000008:00100000:0.0:1789760794.242686:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514bb80 - object 0x240000400:5 offset 29282304<>29286400 00000100:00100000:1.0:1789760794.242951:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512779648:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760794.289965:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974468 gen 3 00000008:00010000:1.0:1789760794.290001:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760794.290007:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.290016:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.290044:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760794.290066:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512779648:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760794.290195:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7149, 7149] 00000080:00200000:0.0:1789760794.290209:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.290214:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.290225:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29286400 i_blocks 57351 00000020:00200000:0.0:1789760794.290228:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.290236:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.290261:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29286400, count: 4096 00000080:00200000:0.0:1789760794.290266:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29286400, bytes: 4096 00000080:00200000:0.0:1789760794.290274:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.290281:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.290283:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29286400, 29290496) 0 0 00000080:00200000:0.0:1789760794.290286:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.290299:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7150, 7150] 00020000:00200000:0.0:1789760794.290305:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29286400 chunk: [29286400, 29290496) 29290496, 4096 00020000:00200000:0.0:1789760794.290308:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.290322:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7150, 7150] 00000008:00010000:0.0:1789760794.290334:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.290338:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760794.290346:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29286400, 29290496) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.290355:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29286400=0x1bee000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.290394:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.290396:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29290496 00000020:00010000:0.0:1789760794.290402:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.290405:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.290408:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.290411:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.290414:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.290430:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.290435:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.290448:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000008:00100000:0.0:1789760794.290514:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149f80 - object 0x240000400:5 offset 29286400<>29290496 00000100:00100000:1.0:1789760794.290650:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760794.290665:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512779776:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760794.323731:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974469 gen 3 00000008:00010000:1.0:1789760794.323768:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760794.323774:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.323783:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.323791:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760794.323814:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512779776:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760794.323979:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7150, 7150] 00000080:00200000:0.0:1789760794.323992:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.323997:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.324009:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29290496 i_blocks 57351 00000020:00200000:0.0:1789760794.324013:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.324021:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.324056:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29290496, count: 4096 00000080:00200000:0.0:1789760794.324062:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29290496, bytes: 4096 00000080:00200000:0.0:1789760794.324070:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.324078:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.324080:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29290496, 29294592) 0 0 00000080:00200000:0.0:1789760794.324083:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.324097:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7151, 7151] 00020000:00200000:0.0:1789760794.324104:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29290496 chunk: [29290496, 29294592) 29294592, 4096 00020000:00200000:0.0:1789760794.324108:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.324123:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7151, 7151] 00000008:00010000:0.0:1789760794.324136:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.324141:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760794.324150:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29290496, 29294592) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.324160:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29290496=0x1bef000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.324201:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.324204:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29294592 00000020:00010000:0.0:1789760794.324211:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.324213:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.324216:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.324219:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.324222:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.324229:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.324235:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.324248:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc780 00000008:00100000:0.0:1789760794.324318:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149500 - object 0x240000400:5 offset 29290496<>29294592 00000100:00100000:1.0:1789760794.324405:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512779904:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760794.355026:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974470 gen 3 00000008:00010000:1.0:1789760794.355071:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760794.355077:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.355087:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.355104:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760794.355123:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512779904:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760794.355259:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7151, 7151] 00000080:00200000:0.0:1789760794.355272:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.355277:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.355289:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29294592 i_blocks 57351 00000020:00200000:0.0:1789760794.355292:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.355300:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.355325:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29294592, count: 4096 00000080:00200000:0.0:1789760794.355331:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29294592, bytes: 4096 00000080:00200000:0.0:1789760794.355338:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.355344:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.355346:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29294592, 29298688) 0 0 00000080:00200000:0.0:1789760794.355350:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.355363:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7152, 7152] 00020000:00200000:0.0:1789760794.355370:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29294592 chunk: [29294592, 29298688) 29298688, 4096 00020000:00200000:0.0:1789760794.355374:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.355388:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7152, 7152] 00000008:00010000:0.0:1789760794.355400:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.355404:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760794.355413:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29294592, 29298688) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.355433:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29294592=0x1bf0000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.355472:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.355474:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29298688 00000020:00010000:0.0:1789760794.355479:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.355482:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.355485:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.355488:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.355491:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.355496:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.355501:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.355512:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccd80 00000008:00100000:0.0:1789760794.355582:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b100 - object 0x240000400:5 offset 29294592<>29298688 00000100:00100000:1.0:1789760794.355660:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760794.355673:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512780032:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760794.389773:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760794.389784:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.389797:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.389815:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760794.389836:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512780032:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760794.390208:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7152, 7152] 00000080:00200000:0.0:1789760794.390221:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.390226:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.390237:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29298688 i_blocks 57351 00000020:00200000:0.0:1789760794.390240:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.390248:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.390274:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29298688, count: 4096 00000080:00200000:0.0:1789760794.390279:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29298688, bytes: 4096 00000080:00200000:0.0:1789760794.390287:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.390293:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.390295:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29298688, 29302784) 0 0 00000080:00200000:0.0:1789760794.390299:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.390311:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7153, 7153] 00020000:00200000:0.0:1789760794.390318:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29298688 chunk: [29298688, 29302784) 29302784, 4096 00020000:00200000:0.0:1789760794.390322:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.390336:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7153, 7153] 00000008:00010000:0.0:1789760794.390348:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.390352:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760794.390361:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29298688, 29302784) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.390371:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29298688=0x1bf1000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.390409:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.390412:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29302784 00000020:00010000:0.0:1789760794.390428:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.390431:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.390435:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.390438:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.390440:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.390446:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.390451:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.390463:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc180 00000008:00100000:0.0:1789760794.390530:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149500 - object 0x240000400:5 offset 29298688<>29302784 00000100:00100000:1.0:1789760794.390721:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512780160:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760794.398096:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760794.426489:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974472 gen 3 00000008:00010000:1.0:1789760794.426531:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760794.426537:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.426545:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.426564:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760794.426586:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512780160:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760794.426869:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7153, 7153] 00000080:00200000:0.0:1789760794.426882:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.426886:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.426919:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29302784 i_blocks 57351 00000020:00200000:0.0:1789760794.426922:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.426930:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.426952:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29302784, count: 4096 00000080:00200000:0.0:1789760794.426957:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29302784, bytes: 4096 00000080:00200000:0.0:1789760794.426965:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.426971:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.426974:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29302784, 29306880) 0 0 00000080:00200000:0.0:1789760794.426977:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.426989:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7154, 7154] 00020000:00200000:0.0:1789760794.426996:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29302784 chunk: [29302784, 29306880) 29306880, 4096 00020000:00200000:0.0:1789760794.427000:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.427014:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7154, 7154] 00000008:00010000:0.0:1789760794.427036:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.427041:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760794.427049:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29302784, 29306880) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.427060:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29302784=0x1bf2000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.427114:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.427116:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29306880 00000020:00010000:0.0:1789760794.427122:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.427125:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.427127:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.427130:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.427133:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.427139:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.427144:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.427155:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000008:00100000:0.0:1789760794.427225:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149f80 - object 0x240000400:5 offset 29302784<>29306880 00000100:00100000:1.0:1789760794.427433:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760794.427447:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512780288:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760794.463841:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974473 gen 3 00000008:00010000:1.0:1789760794.463877:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760794.463883:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.463892:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.463927:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760794.463946:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512780288:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760794.464229:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7154, 7154] 00000080:00200000:0.0:1789760794.464241:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.464246:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.464257:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29306880 i_blocks 57351 00000020:00200000:0.0:1789760794.464260:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.464268:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.464294:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29306880, count: 4096 00000080:00200000:0.0:1789760794.464300:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29306880, bytes: 4096 00000080:00200000:0.0:1789760794.464307:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.464314:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.464316:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29306880, 29310976) 0 0 00000080:00200000:0.0:1789760794.464319:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.464331:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7155, 7155] 00020000:00200000:0.0:1789760794.464337:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29306880 chunk: [29306880, 29310976) 29310976, 4096 00020000:00200000:0.0:1789760794.464341:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.464354:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7155, 7155] 00000008:00010000:0.0:1789760794.464366:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.464370:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760794.464379:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29306880, 29310976) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.464389:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29306880=0x1bf3000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.464484:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.464486:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29310976 00000020:00010000:0.0:1789760794.464494:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.464497:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.464500:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.464503:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.464506:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.464512:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.464517:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.464533:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccc00 00000008:00100000:0.0:1789760794.464609:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514bb80 - object 0x240000400:5 offset 29306880<>29310976 00000100:00100000:1.0:1789760794.464668:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512780416:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760794.495647:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974474 gen 3 00000008:00010000:1.0:1789760794.495682:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760794.495689:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.495706:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.495725:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760794.495750:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512780416:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760794.496011:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7155, 7155] 00000080:00200000:0.0:1789760794.496023:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.496037:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.496048:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29310976 i_blocks 57351 00000020:00200000:0.0:1789760794.496051:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.496059:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.496085:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29310976, count: 4096 00000080:00200000:0.0:1789760794.496091:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29310976, bytes: 4096 00000080:00200000:0.0:1789760794.496098:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.496105:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.496108:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29310976, 29315072) 0 0 00000080:00200000:0.0:1789760794.496111:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.496123:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7156, 7156] 00020000:00200000:0.0:1789760794.496129:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29310976 chunk: [29310976, 29315072) 29315072, 4096 00020000:00200000:0.0:1789760794.496133:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.496148:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7156, 7156] 00000008:00010000:0.0:1789760794.496161:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.496166:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760794.496174:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29310976, 29315072) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.496185:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29310976=0x1bf4000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.496230:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.496233:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29315072 00000020:00010000:0.0:1789760794.496240:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.496243:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.496245:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.496248:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.496250:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.496257:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.496262:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.496277:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cce40 00000008:00100000:0.0:1789760794.496359:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149c00 - object 0x240000400:5 offset 29310976<>29315072 00000100:00100000:1.0:1789760794.496451:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760794.496464:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512780544:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760794.530160:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974475 gen 3 00000008:00010000:1.0:1789760794.530193:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760794.530199:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.530209:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.530227:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760794.530250:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512780544:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760794.530407:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7156, 7156] 00000080:00200000:0.0:1789760794.530468:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.530473:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.530484:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29315072 i_blocks 57351 00000020:00200000:0.0:1789760794.530487:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.530494:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.530519:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29315072, count: 4096 00000080:00200000:0.0:1789760794.530525:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29315072, bytes: 4096 00000080:00200000:0.0:1789760794.530532:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.530540:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.530542:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29315072, 29319168) 0 0 00000080:00200000:0.0:1789760794.530545:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.530559:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7157, 7157] 00020000:00200000:0.0:1789760794.530565:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29315072 chunk: [29315072, 29319168) 29319168, 4096 00020000:00200000:0.0:1789760794.530569:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.530582:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7157, 7157] 00000008:00010000:0.0:1789760794.530595:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.530600:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760794.530609:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29315072, 29319168) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.530619:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29315072=0x1bf5000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.530663:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.530666:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29319168 00000020:00010000:0.0:1789760794.530673:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.530675:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.530678:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.530681:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.530683:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.530690:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.530695:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.530709:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cce40 00000008:00100000:0.0:1789760794.530783:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148000 - object 0x240000400:5 offset 29315072<>29319168 00000100:00100000:1.0:1789760794.530916:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760794.530929:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512780672:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760794.563952:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974476 gen 3 00000008:00010000:1.0:1789760794.563984:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cce40 00000020:00200000:1.0:1789760794.563989:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.564066:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.564083:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cce40 00000100:00100000:1.0:1789760794.564105:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512780672:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760794.564242:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7157, 7157] 00000080:00200000:0.0:1789760794.564256:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.564260:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.564272:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29319168 i_blocks 57351 00000020:00200000:0.0:1789760794.564275:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.564283:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.564307:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29319168, count: 4096 00000080:00200000:0.0:1789760794.564313:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29319168, bytes: 4096 00000080:00200000:0.0:1789760794.564321:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.564327:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.564329:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29319168, 29323264) 0 0 00000080:00200000:0.0:1789760794.564332:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.564345:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7158, 7158] 00020000:00200000:0.0:1789760794.564351:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29319168 chunk: [29319168, 29323264) 29323264, 4096 00020000:00200000:0.0:1789760794.564355:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.564369:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7158, 7158] 00000008:00010000:0.0:1789760794.564380:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.564385:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760794.564394:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29319168, 29323264) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.564404:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29319168=0x1bf6000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.564458:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.564461:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29323264 00000020:00010000:0.0:1789760794.564467:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.564470:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.564473:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.564476:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.564479:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.564484:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.564488:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.564500:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccc00 00000008:00100000:0.0:1789760794.564579:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149180 - object 0x240000400:5 offset 29319168<>29323264 00000100:00100000:1.0:1789760794.564717:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760794.564731:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512780800:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760794.597035:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974477 gen 3 00000008:00010000:1.0:1789760794.597067:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccc00 00000020:00200000:1.0:1789760794.597073:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.597082:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.597100:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccc00 00000100:00100000:1.0:1789760794.597121:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512780800:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760794.597363:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7158, 7158] 00000080:00200000:0.0:1789760794.597376:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.597380:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.597392:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29323264 i_blocks 57351 00000020:00200000:0.0:1789760794.597395:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.597403:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.597482:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29323264, count: 4096 00000080:00200000:0.0:1789760794.597488:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29323264, bytes: 4096 00000080:00200000:0.0:1789760794.597495:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.597502:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.597504:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29323264, 29327360) 0 0 00000080:00200000:0.0:1789760794.597566:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.597579:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7159, 7159] 00020000:00200000:0.0:1789760794.597585:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29323264 chunk: [29323264, 29327360) 29327360, 4096 00020000:00200000:0.0:1789760794.597588:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.597603:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7159, 7159] 00000008:00010000:0.0:1789760794.597617:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.597621:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760794.597630:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29323264, 29327360) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.597640:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29323264=0x1bf7000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.597682:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.597685:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29327360 00000020:00010000:0.0:1789760794.597692:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.597695:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.597698:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.597712:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.597714:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.597721:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.597727:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.597742:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000008:00100000:0.0:1789760794.597888:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b480 - object 0x240000400:5 offset 29323264<>29327360 00000100:00100000:1.0:1789760794.597947:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512780928:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760794.652278:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974478 gen 3 00000008:00010000:1.0:1789760794.652318:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000020:00200000:1.0:1789760794.652324:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.652333:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.652357:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc0c0 00000100:00100000:1.0:1789760794.652382:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512780928:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760794.652488:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7159, 7159] 00000080:00200000:0.0:1789760794.652502:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.652507:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.652518:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29327360 i_blocks 57351 00000020:00200000:0.0:1789760794.652521:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.652529:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.652555:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29327360, count: 4096 00000080:00200000:0.0:1789760794.652562:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29327360, bytes: 4096 00000080:00200000:0.0:1789760794.652569:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.652576:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.652578:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29327360, 29331456) 0 0 00000080:00200000:0.0:1789760794.652582:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.652595:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7160, 7160] 00020000:00200000:0.0:1789760794.652601:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29327360 chunk: [29327360, 29331456) 29331456, 4096 00020000:00200000:0.0:1789760794.652605:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.652619:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7160, 7160] 00000008:00010000:0.0:1789760794.652632:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.652636:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760794.652645:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29327360, 29331456) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.652654:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29327360=0x1bf8000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.652694:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.652696:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29331456 00000020:00010000:0.0:1789760794.652703:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.652706:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.652709:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.652712:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.652715:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.652722:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.652727:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.652742:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc180 00000008:00100000:0.0:1789760794.652815:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514b800 - object 0x240000400:5 offset 29327360<>29331456 00000100:00100000:1.0:1789760794.652998:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512781056:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760794.731490:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974479 gen 3 00000008:00010000:1.0:1789760794.731527:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc180 00000020:00200000:1.0:1789760794.731533:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.731543:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.731610:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc180 00000100:00100000:1.0:1789760794.731633:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512781056:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760794.734620:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7160, 7160] 00000080:00200000:3.0:1789760794.734640:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760794.734645:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760794.734659:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29331456 i_blocks 57351 00000020:00200000:3.0:1789760794.734663:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760794.734676:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760794.734700:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29331456, count: 4096 00000080:00200000:3.0:1789760794.734707:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29331456, bytes: 4096 00000080:00200000:3.0:1789760794.734717:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760794.734724:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760794.734726:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29331456, 29335552) 0 0 00000080:00200000:3.0:1789760794.734730:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760794.734743:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7161, 7161] 00020000:00200000:3.0:1789760794.734749:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29331456 chunk: [29331456, 29335552) 29335552, 4096 00020000:00200000:3.0:1789760794.734753:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760794.734768:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7161, 7161] 00000008:00010000:3.0:1789760794.734781:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760794.734785:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760794.734793:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29331456, 29335552) append 0 append lockpos 0 00000080:00200000:3.0:1789760794.734803:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29331456=0x1bf9000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760794.734852:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760794.734855:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29335552 00000020:00010000:3.0:1789760794.734861:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760794.734864:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.734867:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.734871:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760794.734874:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760794.734880:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760794.734885:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760794.734935:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760794.735017:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 29331456<>29335552 00000100:00100000:2.0:1789760794.735749:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512781184:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760794.781095:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974480 gen 3 00000008:00010000:3.0:1789760794.781132:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760794.781138:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760794.781147:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760794.781168:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760794.781190:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512781184:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760794.781238:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7161, 7161] 00000080:00200000:0.0:1789760794.781253:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.781259:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.781272:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29335552 i_blocks 57351 00000020:00200000:0.0:1789760794.781275:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.781284:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.781312:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29335552, count: 4096 00000080:00200000:0.0:1789760794.781319:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29335552, bytes: 4096 00000080:00200000:0.0:1789760794.781327:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.781334:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.781337:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29335552, 29339648) 0 0 00000080:00200000:0.0:1789760794.781341:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.781353:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7162, 7162] 00020000:00200000:0.0:1789760794.781360:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29335552 chunk: [29335552, 29339648) 29339648, 4096 00020000:00200000:0.0:1789760794.781364:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.781395:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7162, 7162] 00000008:00010000:0.0:1789760794.781409:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.781413:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760794.781422:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29335552, 29339648) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.781432:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29335552=0x1bfa000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.781478:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.781481:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29339648 00000020:00010000:0.0:1789760794.781488:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.781491:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.781495:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.781498:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.781501:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.781506:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.781512:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.781527:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccd80 00000008:00100000:0.0:1789760794.781603:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148380 - object 0x240000400:5 offset 29335552<>29339648 00000100:00100000:1.0:1789760794.783423:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760794.783449:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512781312:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760794.857697:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974481 gen 3 00000008:00010000:1.0:1789760794.857733:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccd80 00000020:00200000:1.0:1789760794.857739:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.857749:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.857773:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccd80 00000100:00100000:1.0:1789760794.857797:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512781312:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760794.857929:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7162, 7162] 00000080:00200000:0.0:1789760794.857943:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.857948:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.857961:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29339648 i_blocks 57351 00000020:00200000:0.0:1789760794.857964:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.857973:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.857998:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29339648, count: 4096 00000080:00200000:0.0:1789760794.858003:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29339648, bytes: 4096 00000080:00200000:0.0:1789760794.858012:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.858019:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.858021:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29339648, 29343744) 0 0 00000080:00200000:0.0:1789760794.858034:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.858048:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7163, 7163] 00020000:00200000:0.0:1789760794.858056:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29339648 chunk: [29339648, 29343744) 29343744, 4096 00020000:00200000:0.0:1789760794.858059:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.858075:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7163, 7163] 00000008:00010000:0.0:1789760794.858089:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.858094:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760794.858102:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29339648, 29343744) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.858113:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29339648=0x1bfb000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.858157:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.858160:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29343744 00000020:00010000:0.0:1789760794.858166:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.858170:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.858173:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.858176:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.858179:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.858186:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.858266:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.858282:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc780 00000008:00100000:0.0:1789760794.858357:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005149880 - object 0x240000400:5 offset 29339648<>29343744 00000100:00100000:1.0:1789760794.858485:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760794.858500:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512781440:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760794.939471:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc780 00000020:00200000:1.0:1789760794.939481:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.939490:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.939579:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc780 00000100:00100000:1.0:1789760794.939609:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512781440:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760794.940765:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7163, 7163] 00000080:00200000:0.0:1789760794.940781:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760794.940787:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760794.940799:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29343744 i_blocks 57351 00000020:00200000:0.0:1789760794.940803:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760794.940813:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760794.940839:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29343744, count: 4096 00000080:00200000:0.0:1789760794.940846:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29343744, bytes: 4096 00000080:00200000:0.0:1789760794.940856:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760794.940924:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760794.940927:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29343744, 29347840) 0 0 00000080:00200000:0.0:1789760794.940931:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760794.940944:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7164, 7164] 00020000:00200000:0.0:1789760794.940951:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29343744 chunk: [29343744, 29347840) 29347840, 4096 00020000:00200000:0.0:1789760794.940955:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760794.940971:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7164, 7164] 00000008:00010000:0.0:1789760794.940985:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760794.940989:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760794.940999:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29343744, 29347840) append 0 append lockpos 0 00000080:00200000:0.0:1789760794.941009:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29343744=0x1bfc000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760794.941195:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760794.941198:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29347840 00000020:00010000:0.0:1789760794.941206:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760794.941209:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.941212:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760794.941216:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760794.941218:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760794.941226:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760794.941232:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760794.941254:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000008:00100000:0.0:1789760794.941333:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a300 - object 0x240000400:5 offset 29343744<>29347840 00000100:00100000:1.0:1789760794.941410:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512781568:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760794.993072:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974482 gen 3 00000100:00100000:1.0:1789760794.993104:0:2358:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00514a300 x1876699512781568/t4294974483(4294974483) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760810 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760794.993306:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000020:00200000:1.0:1789760794.993313:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760794.993323:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760794.993344:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc3c0 00000100:00100000:1.0:1789760794.993363:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512781568:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760794.993446:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7164, 7164] 00000080:00200000:3.0:1789760794.993459:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760794.993463:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760794.993474:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29347840 i_blocks 57351 00000020:00200000:3.0:1789760794.993478:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760794.993485:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760794.993522:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29347840, count: 4096 00000080:00200000:3.0:1789760794.993528:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29347840, bytes: 4096 00000080:00200000:3.0:1789760794.993536:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760794.993544:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760794.993546:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29347840, 29351936) 0 0 00000080:00200000:3.0:1789760794.993549:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760794.993562:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7165, 7165] 00020000:00200000:3.0:1789760794.993569:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29347840 chunk: [29347840, 29351936) 29351936, 4096 00020000:00200000:3.0:1789760794.993572:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760794.993587:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7165, 7165] 00000008:00010000:3.0:1789760794.993601:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760794.993607:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760794.993616:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29347840, 29351936) append 0 append lockpos 0 00000080:00200000:3.0:1789760794.993628:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29347840=0x1bfd000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760794.993687:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760794.993690:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29351936 00000020:00010000:3.0:1789760794.993697:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760794.993701:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.993704:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760794.993707:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760794.993710:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760794.993716:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760794.993722:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760794.993735:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760794.993828:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 29347840<>29351936 00000100:00100000:3.0:1789760794.993873:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760794.993886:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512781696:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760795.060684:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974483 gen 3 00000100:00100000:3.0:1789760795.060724:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc0107ace00 x1876699512781696/t4294974484(4294974484) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760810 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760795.060759:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:3.0:1789760795.060765:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.060773:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.060952:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:3.0:1789760795.060971:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512781696:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760795.061051:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7165, 7165] 00000080:00200000:2.0:1789760795.061064:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.061069:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.061079:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29351936 i_blocks 57351 00000020:00200000:2.0:1789760795.061083:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.061090:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.061119:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29351936, count: 4096 00000080:00200000:2.0:1789760795.061126:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29351936, bytes: 4096 00000080:00200000:2.0:1789760795.061134:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.061141:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.061143:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29351936, 29356032) 0 0 00000080:00200000:2.0:1789760795.061147:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.061159:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7166, 7166] 00020000:00200000:2.0:1789760795.061166:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29351936 chunk: [29351936, 29356032) 29356032, 4096 00020000:00200000:2.0:1789760795.061169:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.061185:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7166, 7166] 00000008:00010000:2.0:1789760795.061199:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.061203:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760795.061212:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29351936, 29356032) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.061224:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29351936=0x1bfe000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.061270:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.061273:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29356032 00000020:00010000:2.0:1789760795.061279:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.061283:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.061286:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.061289:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.061292:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.061299:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.061304:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.061318:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760795.061394:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 29351936<>29356032 00000100:00100000:2.0:1789760795.061438:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760795.061450:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512781824:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760795.100756:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974485 gen 3 00000008:00010000:3.0:1789760795.100807:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760795.100814:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.100823:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.100880:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760795.100948:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512781824:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760795.101153:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7166, 7166] 00000080:00200000:2.0:1789760795.101166:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.101170:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.101181:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29356032 i_blocks 57351 00000020:00200000:2.0:1789760795.101184:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.101191:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.101216:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29356032, count: 4096 00000080:00200000:2.0:1789760795.101222:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29356032, bytes: 4096 00000080:00200000:2.0:1789760795.101231:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.101237:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.101239:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29356032, 29360128) 0 0 00000080:00200000:2.0:1789760795.101243:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.101256:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7167, 7167] 00020000:00200000:2.0:1789760795.101263:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29356032 chunk: [29356032, 29360128) 29360128, 4096 00020000:00200000:2.0:1789760795.101267:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.101281:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7167, 7167] 00000008:00010000:2.0:1789760795.101294:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.101299:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760795.101308:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29356032, 29360128) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.101318:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29356032=0x1bff000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.101560:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.101563:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29360128 00000020:00010000:2.0:1789760795.101571:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.101574:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.101577:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.101580:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.101583:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.101590:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.101596:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.101611:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760795.101760:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 29356032<>29360128 00000100:00100000:3.0:1789760795.101820:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512781952:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760795.138605:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974486 gen 3 00000008:00010000:3.0:1789760795.138641:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760795.138648:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.138656:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.138723:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760795.138749:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512781952:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760795.139050:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7167, 7167] 00000080:00200000:2.0:1789760795.139062:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.139067:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.139078:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29360128 i_blocks 57351 00000020:00200000:2.0:1789760795.139082:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.139090:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.139117:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29360128, count: 4096 00000080:00200000:2.0:1789760795.139122:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29360128, bytes: 4096 00000080:00200000:2.0:1789760795.139132:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.139138:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.139141:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29360128, 29364224) 0 0 00000080:00200000:2.0:1789760795.139144:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.139156:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7168, 7168] 00020000:00200000:2.0:1789760795.139172:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29360128 chunk: [29360128, 29364224) 29364224, 4096 00020000:00200000:2.0:1789760795.139176:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.139192:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7168, 7168] 00000008:00010000:2.0:1789760795.139207:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.139211:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760795.139221:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29360128, 29364224) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.139232:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29360128=0x1c00000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.139277:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.139280:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29364224 00000020:00010000:2.0:1789760795.139286:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.139289:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.139292:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.139295:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.139298:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.139304:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.139310:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.139324:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760795.139401:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 29360128<>29364224 00000100:00100000:3.0:1789760795.139624:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512782080:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760795.176135:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974487 gen 3 00000008:00010000:3.0:1789760795.176169:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760795.176175:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.176183:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.176200:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760795.176225:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512782080:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760795.176305:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7168, 7168] 00000080:00200000:2.0:1789760795.176318:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.176323:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.176334:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29364224 i_blocks 57351 00000020:00200000:2.0:1789760795.176337:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.176344:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.176369:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29364224, count: 4096 00000080:00200000:2.0:1789760795.176375:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29364224, bytes: 4096 00000080:00200000:2.0:1789760795.176382:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.176389:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.176391:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29364224, 29368320) 0 0 00000080:00200000:2.0:1789760795.176395:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.176408:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7169, 7169] 00020000:00200000:2.0:1789760795.176424:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29364224 chunk: [29364224, 29368320) 29368320, 4096 00020000:00200000:2.0:1789760795.176428:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.176442:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7169, 7169] 00000008:00010000:2.0:1789760795.176456:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.176461:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760795.176470:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29364224, 29368320) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.176479:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29364224=0x1c01000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.176519:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.176531:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29368320 00000020:00010000:2.0:1789760795.176538:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.176541:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.176544:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.176548:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.176550:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.176557:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.176563:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.176575:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760795.176651:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368dc00 - object 0x240000400:5 offset 29364224<>29368320 00000100:00100000:3.0:1789760795.178842:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760795.178874:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512782208:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760795.218575:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974488 gen 3 00000008:00010000:3.0:1789760795.218609:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760795.218614:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.218623:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.218640:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760795.218662:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512782208:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760795.218799:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7169, 7169] 00000080:00200000:2.0:1789760795.218811:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.218816:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.218828:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29368320 i_blocks 59399 00000020:00200000:2.0:1789760795.218831:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.218838:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.218864:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29368320, count: 4096 00000080:00200000:2.0:1789760795.218870:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29368320, bytes: 4096 00000080:00200000:2.0:1789760795.218878:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.218885:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.218887:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29368320, 29372416) 0 0 00000080:00200000:2.0:1789760795.218891:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.218930:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7170, 7170] 00020000:00200000:2.0:1789760795.218937:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29368320 chunk: [29368320, 29372416) 29372416, 4096 00020000:00200000:2.0:1789760795.218940:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.218954:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7170, 7170] 00000008:00010000:2.0:1789760795.218968:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.218972:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760795.218980:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29368320, 29372416) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.218991:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29368320=0x1c02000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.219041:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.219052:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29372416 00000020:00010000:2.0:1789760795.219059:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.219062:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.219065:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.219068:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.219071:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.219078:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.219083:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.219097:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760795.219170:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c700 - object 0x240000400:5 offset 29368320<>29372416 00000100:00100000:3.0:1789760795.219304:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512782336:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760795.264931:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974489 gen 3 00000008:00010000:3.0:1789760795.264968:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760795.264974:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.264983:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.265145:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760795.265169:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512782336:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760795.265241:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7170, 7170] 00000080:00200000:2.0:1789760795.265254:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.265258:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.265269:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29372416 i_blocks 59399 00000020:00200000:2.0:1789760795.265272:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.265279:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.265306:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29372416, count: 4096 00000080:00200000:2.0:1789760795.265313:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29372416, bytes: 4096 00000080:00200000:2.0:1789760795.265321:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.265327:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.265329:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29372416, 29376512) 0 0 00000080:00200000:2.0:1789760795.265333:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.265346:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7171, 7171] 00020000:00200000:2.0:1789760795.265352:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29372416 chunk: [29372416, 29376512) 29376512, 4096 00020000:00200000:2.0:1789760795.265355:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.265370:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7171, 7171] 00000008:00010000:2.0:1789760795.265383:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.265387:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760795.265396:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29372416, 29376512) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.265405:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29372416=0x1c03000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.265682:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.265685:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29376512 00000020:00010000:2.0:1789760795.265692:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.265696:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.265699:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.265702:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.265705:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.265712:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.265717:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.265730:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760795.265808:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d180 - object 0x240000400:5 offset 29372416<>29376512 00000100:00100000:3.0:1789760795.265867:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512782464:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760795.289877:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974490 gen 3 00000008:00010000:3.0:1789760795.289938:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760795.289944:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.289953:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.289971:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760795.289994:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512782464:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760795.290092:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7171, 7171] 00000080:00200000:2.0:1789760795.290104:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.290110:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.290122:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29376512 i_blocks 59399 00000020:00200000:2.0:1789760795.290125:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.290132:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.290155:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29376512, count: 4096 00000080:00200000:2.0:1789760795.290161:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29376512, bytes: 4096 00000080:00200000:2.0:1789760795.290168:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.290173:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.290175:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29376512, 29380608) 0 0 00000080:00200000:2.0:1789760795.290179:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.290191:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7172, 7172] 00020000:00200000:2.0:1789760795.290198:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29376512 chunk: [29376512, 29380608) 29380608, 4096 00020000:00200000:2.0:1789760795.290201:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.290216:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7172, 7172] 00000008:00010000:2.0:1789760795.290228:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.290233:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760795.290242:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29376512, 29380608) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.290252:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29376512=0x1c04000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.290292:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.290295:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29380608 00000020:00010000:2.0:1789760795.290301:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.290304:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.290307:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.290310:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.290313:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.290319:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.290324:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.290337:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760795.290408:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e680 - object 0x240000400:5 offset 29376512<>29380608 00000100:00100000:3.0:1789760795.290651:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512782592:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760795.327978:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974491 gen 3 00000008:00010000:3.0:1789760795.328014:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760795.328028:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.328038:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.328089:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760795.328111:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512782592:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760795.328172:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7172, 7172] 00000080:00200000:2.0:1789760795.328184:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.328190:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.328203:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29380608 i_blocks 59399 00000020:00200000:2.0:1789760795.328206:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.328214:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.328239:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29380608, count: 4096 00000080:00200000:2.0:1789760795.328245:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29380608, bytes: 4096 00000080:00200000:2.0:1789760795.328252:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.328260:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.328261:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29380608, 29384704) 0 0 00000080:00200000:2.0:1789760795.328265:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.328277:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7173, 7173] 00020000:00200000:2.0:1789760795.328284:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29380608 chunk: [29380608, 29384704) 29384704, 4096 00020000:00200000:2.0:1789760795.328288:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.328303:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7173, 7173] 00000008:00010000:2.0:1789760795.328316:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.328321:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760795.328329:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29380608, 29384704) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.328340:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29380608=0x1c05000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.328379:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.328381:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29384704 00000020:00010000:2.0:1789760795.328388:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.328391:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.328394:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.328397:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.328401:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.328407:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.328413:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.328427:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760795.328566:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368df80 - object 0x240000400:5 offset 29380608<>29384704 00000100:00100000:3.0:1789760795.328619:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512782720:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760795.356210:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974492 gen 3 00000008:00010000:3.0:1789760795.356245:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760795.356251:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.356260:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.356305:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760795.356327:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512782720:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760795.356389:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7173, 7173] 00000080:00200000:2.0:1789760795.356400:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.356404:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.356415:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29384704 i_blocks 59399 00000020:00200000:2.0:1789760795.356470:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.356478:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.356502:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29384704, count: 4096 00000080:00200000:2.0:1789760795.356507:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29384704, bytes: 4096 00000080:00200000:2.0:1789760795.356514:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.356520:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.356522:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29384704, 29388800) 0 0 00000080:00200000:2.0:1789760795.356525:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.356537:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7174, 7174] 00020000:00200000:2.0:1789760795.356543:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29384704 chunk: [29384704, 29388800) 29388800, 4096 00020000:00200000:2.0:1789760795.356546:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.356559:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7174, 7174] 00000008:00010000:2.0:1789760795.356570:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.356574:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760795.356582:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29384704, 29388800) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.356592:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29384704=0x1c06000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.356630:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.356632:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29388800 00000020:00010000:2.0:1789760795.356637:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.356640:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.356643:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.356646:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.356648:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.356653:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.356657:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.356669:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760795.357161:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368e300 - object 0x240000400:5 offset 29384704<>29388800 00000100:00100000:3.0:1789760795.357244:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512782848:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760795.391987:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974493 gen 3 00000008:00010000:3.0:1789760795.392045:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760795.392052:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.392060:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.392082:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000080:00200000:2.0:1789760795.392125:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7174, 7174] 00000080:00200000:2.0:1789760795.392138:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.392143:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.392154:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29388800 i_blocks 59399 00000020:00200000:2.0:1789760795.392157:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.392166:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.392190:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29388800, count: 4096 00000080:00200000:2.0:1789760795.392207:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29388800, bytes: 4096 00000080:00200000:2.0:1789760795.392218:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.392225:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.392227:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29388800, 29392896) 0 0 00000080:00200000:2.0:1789760795.392231:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.392244:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7175, 7175] 00020000:00200000:2.0:1789760795.392259:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29388800 chunk: [29388800, 29392896) 29392896, 4096 00020000:00200000:2.0:1789760795.392264:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.392279:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7175, 7175] 00000008:00010000:2.0:1789760795.392292:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.392297:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760795.392443:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29388800, 29392896) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.392454:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29388800=0x1c07000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.392491:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.392493:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29392896 00000020:00010000:2.0:1789760795.392499:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.392503:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.392505:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.392507:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.392510:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.392515:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.392520:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1789760795.392763:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512782848:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760795.392799:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:3.0:1789760795.392893:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 29388800<>29392896 00000100:00100000:2.0:1789760795.394566:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760795.395005:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512782976:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760795.421234:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760795.423226:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512783232:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760795.423321:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1789760795.423329:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e6cf7267-9b19-462a-a263-14115df15dab:2360:1876699512783104:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760795.423372:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2360:1876699512783360:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760795.434605:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512783232:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760795.435042:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:e6cf7267-9b19-462a-a263-14115df15dab:2360:1876699512783104:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760795.435055:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2360:1876699512783360:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760795.445735:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974494 gen 3 00000008:00010000:3.0:1789760795.445772:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760795.445779:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.445786:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.445803:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760795.445832:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512782976:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760795.446104:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7175, 7175] 00000080:00200000:2.0:1789760795.446116:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.446121:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.446133:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29392896 i_blocks 59399 00000020:00200000:2.0:1789760795.446136:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.446144:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.446167:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29392896, count: 4096 00000080:00200000:2.0:1789760795.446173:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29392896, bytes: 4096 00000080:00200000:2.0:1789760795.446181:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.446188:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.446191:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29392896, 29396992) 0 0 00000080:00200000:2.0:1789760795.446194:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.446206:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7176, 7176] 00020000:00200000:2.0:1789760795.446212:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29392896 chunk: [29392896, 29396992) 29396992, 4096 00020000:00200000:2.0:1789760795.446216:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.446229:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7176, 7176] 00000008:00010000:2.0:1789760795.446242:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.446247:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760795.446256:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29392896, 29396992) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.446266:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29392896=0x1c08000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.446306:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.446309:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29396992 00000020:00010000:2.0:1789760795.446315:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.446319:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.446322:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.446325:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.446327:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.446335:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.446340:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.446352:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9dd80 00000008:00100000:2.0:1789760795.446766:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ed80 - object 0x240000400:5 offset 29392896<>29396992 00000100:00100000:3.0:1789760795.447138:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512783488:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760795.501231:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974495 gen 3 00000008:00010000:3.0:1789760795.501262:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9dd80 00000020:00200000:3.0:1789760795.501267:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.501273:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.501316:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9dd80 00000100:00100000:3.0:1789760795.501334:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512783488:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760795.501440:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7176, 7176] 00000080:00200000:2.0:1789760795.501453:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.501456:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.501466:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29396992 i_blocks 59399 00000020:00200000:2.0:1789760795.501469:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.501475:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.501613:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29396992, count: 4096 00000080:00200000:2.0:1789760795.501619:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29396992, bytes: 4096 00000080:00200000:2.0:1789760795.501625:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.501630:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.501632:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29396992, 29401088) 0 0 00000080:00200000:2.0:1789760795.501635:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.501647:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7177, 7177] 00020000:00200000:2.0:1789760795.501653:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29396992 chunk: [29396992, 29401088) 29401088, 4096 00020000:00200000:2.0:1789760795.501655:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.501668:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7177, 7177] 00000008:00010000:2.0:1789760795.501678:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.501682:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760795.501689:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29396992, 29401088) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.501699:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29396992=0x1c09000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.501745:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.501747:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29401088 00000020:00010000:2.0:1789760795.501754:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.501757:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.501759:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.501763:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.501765:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.501771:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.501777:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.501791:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760795.501872:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f800 - object 0x240000400:5 offset 29396992<>29401088 00000100:00100000:3.0:1789760795.501930:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512783616:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760795.529501:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974496 gen 3 00000008:00010000:3.0:1789760795.529537:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d780 00000020:00200000:3.0:1789760795.529543:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.529551:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.529615:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d780 00000100:00100000:3.0:1789760795.529641:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512783616:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760795.529998:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7177, 7177] 00000080:00200000:2.0:1789760795.530011:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.530016:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.530037:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29401088 i_blocks 59399 00000020:00200000:2.0:1789760795.530040:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.530050:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.530076:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29401088, count: 4096 00000080:00200000:2.0:1789760795.530082:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29401088, bytes: 4096 00000080:00200000:2.0:1789760795.530090:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.530097:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.530098:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29401088, 29405184) 0 0 00000080:00200000:2.0:1789760795.530103:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.530115:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7178, 7178] 00020000:00200000:2.0:1789760795.530122:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29401088 chunk: [29401088, 29405184) 29405184, 4096 00020000:00200000:2.0:1789760795.530126:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.530141:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7178, 7178] 00000008:00010000:2.0:1789760795.530154:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.530159:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760795.530169:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29401088, 29405184) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.530180:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29401088=0x1c0a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.530222:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.530225:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29405184 00000020:00010000:2.0:1789760795.530231:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.530235:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.530238:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.530241:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.530243:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.530250:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.530256:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.530271:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760795.530349:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 29401088<>29405184 00000100:00100000:3.0:1789760795.530422:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512783744:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760795.562182:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974497 gen 3 00000008:00010000:3.0:1789760795.562218:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760795.562224:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.562233:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.562310:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760795.562334:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512783744:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760795.562407:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7178, 7178] 00000080:00200000:2.0:1789760795.562420:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.562424:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.562435:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29405184 i_blocks 59399 00000020:00200000:2.0:1789760795.562439:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.562447:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.562472:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29405184, count: 4096 00000080:00200000:2.0:1789760795.562479:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29405184, bytes: 4096 00000080:00200000:2.0:1789760795.562489:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.562501:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.562503:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29405184, 29409280) 0 0 00000080:00200000:2.0:1789760795.562507:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.562520:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7179, 7179] 00020000:00200000:2.0:1789760795.562526:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29405184 chunk: [29405184, 29409280) 29409280, 4096 00020000:00200000:2.0:1789760795.562530:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.562555:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7179, 7179] 00000008:00010000:2.0:1789760795.562570:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.562574:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760795.562582:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29405184, 29409280) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.562593:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29405184=0x1c0b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.562639:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.562641:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29409280 00000020:00010000:2.0:1789760795.562647:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.562651:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.562653:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.562656:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.562658:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.562664:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.562669:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.562682:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760795.563108:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 29405184<>29409280 00000100:00100000:3.0:1789760795.563167:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512783872:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760795.602030:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974498 gen 3 00000008:00010000:3.0:1789760795.602068:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760795.602075:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.602084:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.602101:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760795.602126:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512783872:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760795.602210:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7179, 7179] 00000080:00200000:2.0:1789760795.602223:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.602227:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.602239:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29409280 i_blocks 59399 00000020:00200000:2.0:1789760795.602242:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.602250:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.602279:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29409280, count: 4096 00000080:00200000:2.0:1789760795.602285:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29409280, bytes: 4096 00000080:00200000:2.0:1789760795.602292:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.602299:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.602300:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29409280, 29413376) 0 0 00000080:00200000:2.0:1789760795.602304:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.602318:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7180, 7180] 00020000:00200000:2.0:1789760795.602325:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29409280 chunk: [29409280, 29413376) 29413376, 4096 00020000:00200000:2.0:1789760795.602329:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.602344:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7180, 7180] 00000008:00010000:2.0:1789760795.602357:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.602362:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760795.602370:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29409280, 29413376) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.602381:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29409280=0x1c0c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.602423:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.602426:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29413376 00000020:00010000:2.0:1789760795.602432:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.602435:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.602438:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.602441:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.602444:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.602450:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.602456:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.602469:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760795.602551:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 29409280<>29413376 00000100:00100000:3.0:1789760795.602701:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512784000:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760795.667149:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974499 gen 3 00000008:00010000:3.0:1789760795.667200:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760795.667207:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.667216:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.667233:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760795.667256:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512784000:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760795.667395:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7180, 7180] 00000080:00200000:2.0:1789760795.667409:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.667425:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.667438:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29413376 i_blocks 59399 00000020:00200000:2.0:1789760795.667441:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.667449:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.667474:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29413376, count: 4096 00000080:00200000:2.0:1789760795.667480:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29413376, bytes: 4096 00000080:00200000:2.0:1789760795.667488:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.667495:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.667497:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29413376, 29417472) 0 0 00000080:00200000:2.0:1789760795.667501:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.667515:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7181, 7181] 00020000:00200000:2.0:1789760795.667522:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29413376 chunk: [29413376, 29417472) 29417472, 4096 00020000:00200000:2.0:1789760795.667525:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.667540:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7181, 7181] 00000008:00010000:2.0:1789760795.667554:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.667559:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760795.667568:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29413376, 29417472) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.667580:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29413376=0x1c0d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.667626:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.667629:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29417472 00000020:00010000:2.0:1789760795.667636:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.667639:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.667642:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.667646:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.667648:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.667654:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.667661:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.667675:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760795.667752:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 29413376<>29417472 00000100:00100000:3.0:1789760795.668105:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760795.668119:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512784128:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760795.704725:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974500 gen 3 00000008:00010000:3.0:1789760795.704838:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760795.704845:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.704853:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.704937:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760795.704960:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512784128:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760795.707588:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7181, 7181] 00000080:00200000:2.0:1789760795.707608:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.707614:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.707628:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29417472 i_blocks 59399 00000020:00200000:2.0:1789760795.707633:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.707645:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.707673:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29417472, count: 4096 00000080:00200000:2.0:1789760795.707681:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29417472, bytes: 4096 00000080:00200000:2.0:1789760795.707692:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.707699:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.707701:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29417472, 29421568) 0 0 00000080:00200000:2.0:1789760795.707705:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.707717:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7182, 7182] 00020000:00200000:2.0:1789760795.707725:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29417472 chunk: [29417472, 29421568) 29421568, 4096 00020000:00200000:2.0:1789760795.707728:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.707746:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7182, 7182] 00000008:00010000:2.0:1789760795.707771:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.707776:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760795.707786:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29417472, 29421568) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.707797:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29417472=0x1c0e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.707850:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.707853:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29421568 00000020:00010000:2.0:1789760795.707860:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.707863:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.707866:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.707869:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.707872:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.707879:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.707884:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.707938:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760795.708033:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 29417472<>29421568 00000100:00100000:3.0:1789760795.709683:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760795.709711:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512784256:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760795.748974:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974501 gen 3 00000008:00010000:3.0:1789760795.749011:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760795.749027:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.749036:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.749053:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760795.749077:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512784256:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760795.749266:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7182, 7182] 00000080:00200000:2.0:1789760795.749279:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.749284:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.749296:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29421568 i_blocks 59399 00000020:00200000:2.0:1789760795.749299:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.749307:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.749333:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29421568, count: 4096 00000080:00200000:2.0:1789760795.749339:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29421568, bytes: 4096 00000080:00200000:2.0:1789760795.749346:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.749354:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.749355:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29421568, 29425664) 0 0 00000080:00200000:2.0:1789760795.749359:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.749372:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7183, 7183] 00020000:00200000:2.0:1789760795.749379:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29421568 chunk: [29421568, 29425664) 29425664, 4096 00020000:00200000:2.0:1789760795.749383:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.749399:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7183, 7183] 00000008:00010000:2.0:1789760795.749425:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.749429:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760795.749439:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29421568, 29425664) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.749449:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29421568=0x1c0f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.749493:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.749495:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29425664 00000020:00010000:2.0:1789760795.749502:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.749505:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.749507:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.749510:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.749513:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.749519:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.749524:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.749537:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:2.0:1789760795.749615:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 29421568<>29425664 00000100:00100000:3.0:1789760795.749712:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760795.749726:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512784384:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760795.788756:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760795.788768:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.788776:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.788802:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760795.788827:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512784384:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760795.788976:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7183, 7183] 00000080:00200000:2.0:1789760795.788989:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.788994:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.789006:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29425664 i_blocks 59399 00000020:00200000:2.0:1789760795.789009:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.789028:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.789052:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29425664, count: 4096 00000080:00200000:2.0:1789760795.789058:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29425664, bytes: 4096 00000080:00200000:2.0:1789760795.789066:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.789073:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.789075:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29425664, 29429760) 0 0 00000080:00200000:2.0:1789760795.789079:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.789092:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7184, 7184] 00020000:00200000:2.0:1789760795.789098:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29425664 chunk: [29425664, 29429760) 29429760, 4096 00020000:00200000:2.0:1789760795.789102:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.789116:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7184, 7184] 00000008:00010000:2.0:1789760795.789130:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.789135:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760795.789143:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29425664, 29429760) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.789153:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29425664=0x1c10000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.789196:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.789199:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29429760 00000020:00010000:2.0:1789760795.789205:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.789208:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.789211:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.789215:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.789217:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.789223:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.789230:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.789245:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760795.789322:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 29425664<>29429760 00000100:00100000:3.0:1789760795.789445:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760795.789460:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512784512:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760795.833492:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974502 gen 3 00000100:00100000:3.0:1789760795.833522:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368fb80 x1876699512784512/t4294974503(4294974503) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760811 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760795.833558:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760795.833564:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.833571:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.833590:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000100:00100000:3.0:1789760795.833610:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512784512:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760795.833936:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7184, 7184] 00000080:00200000:2.0:1789760795.833948:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760795.833953:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760795.833964:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29429760 i_blocks 59399 00000020:00200000:2.0:1789760795.833968:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760795.833976:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760795.834002:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29429760, count: 4096 00000080:00200000:2.0:1789760795.834009:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29429760, bytes: 4096 00000080:00200000:2.0:1789760795.834028:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760795.834035:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760795.834037:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29429760, 29433856) 0 0 00000080:00200000:2.0:1789760795.834116:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760795.834131:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7185, 7185] 00020000:00200000:2.0:1789760795.834137:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29429760 chunk: [29429760, 29433856) 29433856, 4096 00020000:00200000:2.0:1789760795.834141:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760795.834157:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7185, 7185] 00000008:00010000:2.0:1789760795.834172:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760795.834176:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760795.834186:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29429760, 29433856) append 0 append lockpos 0 00000080:00200000:2.0:1789760795.834196:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29429760=0x1c11000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760795.834241:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760795.834244:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29433856 00000020:00010000:2.0:1789760795.834250:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760795.834254:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.834257:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760795.834260:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760795.834263:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760795.834270:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760795.834275:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760795.834289:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760795.834364:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 29429760<>29433856 00000100:00100000:3.0:1789760795.835218:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512784640:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760795.874352:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974503 gen 3 00000100:00100000:3.0:1789760795.874383:0:2361:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00368d880 x1876699512784640/t4294974504(4294974504) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760811 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:3.0:1789760795.874481:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760795.874487:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760795.874495:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760795.874514:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760795.874532:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512784640:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760795.874624:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7185, 7185] 00000080:00200000:0.0:1789760795.874636:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760795.874641:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760795.874652:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29433856 i_blocks 59399 00000020:00200000:0.0:1789760795.874656:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760795.874664:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760795.874750:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29433856, count: 4096 00000080:00200000:0.0:1789760795.874756:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29433856, bytes: 4096 00000080:00200000:0.0:1789760795.874765:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760795.874771:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760795.874773:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29433856, 29437952) 0 0 00000080:00200000:0.0:1789760795.874778:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760795.874851:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7186, 7186] 00020000:00200000:0.0:1789760795.874859:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29433856 chunk: [29433856, 29437952) 29437952, 4096 00020000:00200000:0.0:1789760795.874862:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760795.874877:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7186, 7186] 00000008:00010000:0.0:1789760795.874890:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760795.874895:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760795.874925:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29433856, 29437952) append 0 append lockpos 0 00000080:00200000:0.0:1789760795.874937:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29433856=0x1c12000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760795.874979:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760795.874981:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29437952 00000020:00010000:0.0:1789760795.874987:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760795.874990:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.874993:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.874995:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760795.874998:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760795.875004:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760795.875010:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760795.875033:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc000 00000008:00100000:0.0:1789760795.875321:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514a680 - object 0x240000400:5 offset 29433856<>29437952 00000100:00100000:1.0:1789760795.875815:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512784768:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760795.941092:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974504 gen 3 00000100:00100000:1.0:1789760795.941124:0:2359:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8dc00514a680 x1876699512784768/t4294974505(4294974505) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 0 dl 1789760811 ref 3 fl Rpc:RQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 00000008:00010000:1.0:1789760795.941163:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc000 00000020:00200000:1.0:1789760795.941169:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760795.941178:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760795.941205:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc000 00000100:00100000:1.0:1789760795.941227:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512784768:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760795.941327:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7186, 7186] 00000080:00200000:0.0:1789760795.941339:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760795.941344:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760795.941356:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29437952 i_blocks 59399 00000020:00200000:0.0:1789760795.941360:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760795.941368:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760795.941406:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29437952, count: 4096 00000080:00200000:0.0:1789760795.941414:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29437952, bytes: 4096 00000080:00200000:0.0:1789760795.941421:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760795.941427:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760795.941429:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29437952, 29442048) 0 0 00000080:00200000:0.0:1789760795.941433:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760795.941444:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7187, 7187] 00020000:00200000:0.0:1789760795.941451:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29437952 chunk: [29437952, 29442048) 29442048, 4096 00020000:00200000:0.0:1789760795.941455:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760795.941468:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7187, 7187] 00000008:00010000:0.0:1789760795.941553:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760795.941558:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760795.941568:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29437952, 29442048) append 0 append lockpos 0 00000080:00200000:0.0:1789760795.941578:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29437952=0x1c13000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760795.941622:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760795.941625:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29442048 00000020:00010000:0.0:1789760795.941631:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760795.941634:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.941637:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.941640:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760795.941643:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760795.941649:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760795.941655:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760795.941670:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccf00 00000008:00100000:0.0:1789760795.942600:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148e00 - object 0x240000400:5 offset 29437952<>29442048 00000100:00100000:1.0:1789760795.942846:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1789760795.942868:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512784896:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760795.983603:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974506 gen 3 00000008:00010000:1.0:1789760795.983653:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccf00 00000020:00200000:1.0:1789760795.983660:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760795.983668:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760795.983759:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccf00 00000100:00100000:1.0:1789760795.983783:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512784896:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760795.984425:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7187, 7187] 00000080:00200000:0.0:1789760795.984440:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760795.984446:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760795.984459:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29442048 i_blocks 59399 00000020:00200000:0.0:1789760795.984462:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760795.984471:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760795.984497:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29442048, count: 4096 00000080:00200000:0.0:1789760795.984503:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29442048, bytes: 4096 00000080:00200000:0.0:1789760795.984514:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760795.984521:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760795.984523:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29442048, 29446144) 0 0 00000080:00200000:0.0:1789760795.984527:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760795.984541:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7188, 7188] 00020000:00200000:0.0:1789760795.984548:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29442048 chunk: [29442048, 29446144) 29446144, 4096 00020000:00200000:0.0:1789760795.984552:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760795.984567:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7188, 7188] 00000008:00010000:0.0:1789760795.984581:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760795.984585:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760795.984595:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29442048, 29446144) append 0 append lockpos 0 00000080:00200000:0.0:1789760795.984606:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29442048=0x1c14000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760795.984654:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760795.984657:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29446144 00000020:00010000:0.0:1789760795.984664:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760795.984667:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.984670:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760795.984673:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760795.984676:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760795.984683:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760795.984688:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760795.984706:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cca80 00000008:00100000:0.0:1789760795.985478:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514ad80 - object 0x240000400:5 offset 29442048<>29446144 00000100:00100000:1.0:1789760795.985544:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512785024:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760796.033253:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cca80 00000020:00200000:1.0:1789760796.033264:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760796.033274:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760796.033347:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cca80 00000100:00100000:1.0:1789760796.033372:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512785024:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760796.036942:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7188, 7188] 00000080:00200000:0.0:1789760796.036964:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.036970:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.036984:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29446144 i_blocks 59399 00000020:00200000:0.0:1789760796.036989:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.037002:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.037260:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29446144, count: 4096 00000080:00200000:0.0:1789760796.037270:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29446144, bytes: 4096 00000080:00200000:0.0:1789760796.037282:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.037290:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760796.037293:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29446144, 29450240) 0 0 00000080:00200000:0.0:1789760796.037297:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.037310:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7189, 7189] 00020000:00200000:0.0:1789760796.037317:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29446144 chunk: [29446144, 29450240) 29450240, 4096 00020000:00200000:0.0:1789760796.037321:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.037337:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7189, 7189] 00000008:00010000:0.0:1789760796.037352:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.037357:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760796.037367:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29446144, 29450240) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.037380:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29446144=0x1c15000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.037437:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.037440:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29450240 00000020:00010000:0.0:1789760796.037447:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.037449:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.037452:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.037454:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.037457:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.037463:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.037469:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.037492:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc540 00000008:00100000:0.0:1789760796.037577:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148700 - object 0x240000400:5 offset 29446144<>29450240 00000100:00100000:1.0:1789760796.037687:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512785152:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760796.065379:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974508 gen 3 00000008:00010000:1.0:1789760796.065438:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc540 00000020:00200000:1.0:1789760796.065445:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760796.065453:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760796.065468:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc540 00000100:00100000:1.0:1789760796.065491:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512785152:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760796.065725:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7189, 7189] 00000080:00200000:3.0:1789760796.065739:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.065744:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.065755:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29450240 i_blocks 59399 00000020:00200000:3.0:1789760796.065759:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.065782:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.065814:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29450240, count: 4096 00000080:00200000:3.0:1789760796.065820:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29450240, bytes: 4096 00000080:00200000:3.0:1789760796.065828:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.065835:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.065837:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29450240, 29454336) 0 0 00000080:00200000:3.0:1789760796.065842:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.065854:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7190, 7190] 00020000:00200000:3.0:1789760796.065860:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29450240 chunk: [29450240, 29454336) 29454336, 4096 00020000:00200000:3.0:1789760796.065864:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.065878:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7190, 7190] 00000008:00010000:3.0:1789760796.065891:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.065896:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760796.065933:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29450240, 29454336) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.065944:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29450240=0x1c16000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.065988:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.065991:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29454336 00000020:00010000:3.0:1789760796.065998:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.066001:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.066004:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.066007:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.066011:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.066030:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.066036:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.066049:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760796.066121:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ace00 - object 0x240000400:5 offset 29450240<>29454336 00000100:00100000:3.0:1789760796.066160:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760796.066174:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512785280:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760796.136469:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974509 gen 3 00000008:00010000:2.0:1789760796.136502:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760796.136513:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.136524:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.136589:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760796.136613:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512785280:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760796.136679:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7190, 7190] 00000080:00200000:3.0:1789760796.136759:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.136765:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.136777:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29454336 i_blocks 59399 00000020:00200000:3.0:1789760796.136781:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.136789:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.136813:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29454336, count: 4096 00000080:00200000:3.0:1789760796.136818:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29454336, bytes: 4096 00000080:00200000:3.0:1789760796.136825:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.136832:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.136834:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29454336, 29458432) 0 0 00000080:00200000:3.0:1789760796.136838:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.136850:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7191, 7191] 00020000:00200000:3.0:1789760796.136857:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29454336 chunk: [29454336, 29458432) 29458432, 4096 00020000:00200000:3.0:1789760796.136860:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.136874:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7191, 7191] 00000008:00010000:3.0:1789760796.136887:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.136892:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760796.136927:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29454336, 29458432) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.136938:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29454336=0x1c17000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.136979:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.136982:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29458432 00000020:00010000:3.0:1789760796.136989:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.136992:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.136995:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.136998:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.137001:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.137007:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.137012:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.137035:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760796.137108:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af100 - object 0x240000400:5 offset 29454336<>29458432 00000100:00100000:2.0:1789760796.138256:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760796.138278:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512785408:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760796.175240:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760796.175252:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.175262:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.175276:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000080:00200000:3.0:1789760796.175290:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7191, 7191] 00000100:00100000:2.0:1789760796.175300:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512785408:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760796.175304:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.175308:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.175320:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29458432 i_blocks 59399 00000020:00200000:3.0:1789760796.175323:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.175330:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.175355:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29458432, count: 4096 00000080:00200000:3.0:1789760796.175362:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29458432, bytes: 4096 00000080:00200000:3.0:1789760796.175369:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.175376:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.175378:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29458432, 29462528) 0 0 00000080:00200000:3.0:1789760796.175381:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.175394:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7192, 7192] 00020000:00200000:3.0:1789760796.175399:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29458432 chunk: [29458432, 29462528) 29462528, 4096 00020000:00200000:3.0:1789760796.175403:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.175430:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7192, 7192] 00000008:00010000:3.0:1789760796.175442:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.175446:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760796.175455:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29458432, 29462528) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.175464:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29458432=0x1c18000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.175507:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.175510:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29462528 00000020:00010000:3.0:1789760796.175516:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.175520:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.175523:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.175527:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.175529:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.175535:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.175540:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.175554:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760796.175627:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae680 - object 0x240000400:5 offset 29458432<>29462528 00000100:00100000:2.0:1789760796.175793:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512785536:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760796.218241:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974511 gen 3 00000008:00010000:2.0:1789760796.218287:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760796.218293:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.218301:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.218320:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760796.218341:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512785536:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760796.218626:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7192, 7192] 00000080:00200000:3.0:1789760796.218639:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.218645:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.218669:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29462528 i_blocks 59399 00000020:00200000:3.0:1789760796.218674:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.218682:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.218721:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29462528, count: 4096 00000080:00200000:3.0:1789760796.218728:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29462528, bytes: 4096 00000080:00200000:3.0:1789760796.218748:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.218757:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.218759:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29462528, 29466624) 0 0 00000080:00200000:3.0:1789760796.218763:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.218777:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7193, 7193] 00020000:00200000:3.0:1789760796.218783:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29462528 chunk: [29462528, 29466624) 29466624, 4096 00020000:00200000:3.0:1789760796.218787:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.218802:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7193, 7193] 00000008:00010000:3.0:1789760796.218815:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.218820:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760796.218828:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29462528, 29466624) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.218838:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29462528=0x1c19000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.218878:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.218881:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29466624 00000020:00010000:3.0:1789760796.218888:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.218891:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.218893:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.218897:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.218936:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.218943:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.218949:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.218964:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760796.219054:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af800 - object 0x240000400:5 offset 29462528<>29466624 00000100:00100000:2.0:1789760796.219217:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512785664:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760796.281070:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974512 gen 3 00000008:00010000:2.0:1789760796.281108:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760796.281114:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.282929:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.282959:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760796.282990:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512785664:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760796.284307:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7193, 7193] 00000080:00200000:3.0:1789760796.284324:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.284330:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.284343:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29466624 i_blocks 59399 00000020:00200000:3.0:1789760796.284348:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.284359:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.284385:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29466624, count: 4096 00000080:00200000:3.0:1789760796.284391:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29466624, bytes: 4096 00000080:00200000:3.0:1789760796.284402:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.284410:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.284412:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29466624, 29470720) 0 0 00000080:00200000:3.0:1789760796.284416:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.284428:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7194, 7194] 00020000:00200000:3.0:1789760796.284435:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29466624 chunk: [29466624, 29470720) 29470720, 4096 00020000:00200000:3.0:1789760796.284439:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.284454:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7194, 7194] 00000008:00010000:3.0:1789760796.284467:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.284472:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760796.284480:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29466624, 29470720) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.284490:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29466624=0x1c1a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.284537:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.284540:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29470720 00000020:00010000:3.0:1789760796.284546:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.284549:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.284552:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.284555:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.284558:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.284564:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.284581:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.284599:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eacc0 00000008:00100000:3.0:1789760796.284685:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107adf80 - object 0x240000400:5 offset 29466624<>29470720 00000100:00100000:3.0:1789760796.284735:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760796.284756:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512785792:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760796.317489:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974513 gen 3 00000008:00010000:2.0:1789760796.317535:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eacc0 00000020:00200000:2.0:1789760796.317542:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.317552:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.317569:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eacc0 00000100:00100000:2.0:1789760796.317594:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512785792:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760796.317926:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7194, 7194] 00000080:00200000:3.0:1789760796.317939:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.317944:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.317956:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29470720 i_blocks 59399 00000020:00200000:3.0:1789760796.317959:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.317967:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.317993:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29470720, count: 4096 00000080:00200000:3.0:1789760796.318000:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29470720, bytes: 4096 00000080:00200000:3.0:1789760796.318008:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.318026:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.318028:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29470720, 29474816) 0 0 00000080:00200000:3.0:1789760796.318032:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.318044:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7195, 7195] 00020000:00200000:3.0:1789760796.318051:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29470720 chunk: [29470720, 29474816) 29474816, 4096 00020000:00200000:3.0:1789760796.318055:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.318070:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7195, 7195] 00000008:00010000:3.0:1789760796.318085:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.318090:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760796.318099:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29470720, 29474816) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.318111:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29470720=0x1c1b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.318164:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.318167:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29474816 00000020:00010000:3.0:1789760796.318174:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.318178:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.318181:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.318184:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.318187:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.318192:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.318199:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.318214:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000008:00100000:3.0:1789760796.318290:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aea00 - object 0x240000400:5 offset 29470720<>29474816 00000100:00100000:2.0:1789760796.318930:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760796.318948:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512785920:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760796.387297:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000020:00200000:2.0:1789760796.387309:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.387319:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.387389:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea9c0 00000100:00100000:2.0:1789760796.387419:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512785920:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760796.391315:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7195, 7195] 00000080:00200000:0.0:1789760796.391337:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.391342:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.391356:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29474816 i_blocks 59399 00000020:00200000:0.0:1789760796.391361:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.391374:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.392089:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29474816, count: 4096 00000080:00200000:0.0:1789760796.392100:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29474816, bytes: 4096 00000080:00200000:0.0:1789760796.392113:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.392120:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760796.392122:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29474816, 29478912) 0 0 00000080:00200000:0.0:1789760796.392126:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.392141:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7196, 7196] 00020000:00200000:0.0:1789760796.392148:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29474816 chunk: [29474816, 29478912) 29478912, 4096 00020000:00200000:0.0:1789760796.392151:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.392169:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7196, 7196] 00000008:00010000:0.0:1789760796.392266:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.392272:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760796.392282:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29474816, 29478912) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.392294:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29474816=0x1c1c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.392347:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.392349:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29478912 00000020:00010000:0.0:1789760796.392356:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.392359:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.392362:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.392365:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.392368:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.392375:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.392381:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.392474:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc600 00000008:00100000:0.0:1789760796.392558:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148a80 - object 0x240000400:5 offset 29474816<>29478912 00000100:00100000:1.0:1789760796.392645:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512786048:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760796.439946:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974515 gen 3 00000008:00010000:1.0:1789760796.439990:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc600 00000020:00200000:1.0:1789760796.439996:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760796.440004:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760796.440027:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc600 00000100:00100000:1.0:1789760796.440048:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512786048:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760796.440129:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7196, 7196] 00000080:00200000:3.0:1789760796.440141:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.440146:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.440157:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29478912 i_blocks 59399 00000020:00200000:3.0:1789760796.440160:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.440169:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.440191:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29478912, count: 4096 00000080:00200000:3.0:1789760796.440196:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29478912, bytes: 4096 00000080:00200000:3.0:1789760796.440203:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.440210:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.440213:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29478912, 29483008) 0 0 00000080:00200000:3.0:1789760796.440216:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.440228:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7197, 7197] 00020000:00200000:3.0:1789760796.440234:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29478912 chunk: [29478912, 29483008) 29483008, 4096 00020000:00200000:3.0:1789760796.440238:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.440251:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7197, 7197] 00000008:00010000:3.0:1789760796.440263:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.440267:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760796.440275:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29478912, 29483008) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.440285:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29478912=0x1c1d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.440324:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.440327:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29483008 00000020:00010000:3.0:1789760796.440333:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.440336:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.440339:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.440342:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.440345:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.440352:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.440357:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.440370:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760796.440594:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107aed80 - object 0x240000400:5 offset 29478912<>29483008 00000100:00100000:2.0:1789760796.440667:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512786176:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760796.445992:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760796.479547:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974516 gen 3 00000008:00010000:2.0:1789760796.479583:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760796.479589:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.479597:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.479615:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:2.0:1789760796.479637:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512786176:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760796.479722:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7197, 7197] 00000080:00200000:3.0:1789760796.479736:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.479741:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.479753:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29483008 i_blocks 59399 00000020:00200000:3.0:1789760796.479756:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.479765:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.479855:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29483008, count: 4096 00000080:00200000:3.0:1789760796.479862:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29483008, bytes: 4096 00000080:00200000:3.0:1789760796.479870:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.479876:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.479878:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29483008, 29487104) 0 0 00000080:00200000:3.0:1789760796.479881:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.479894:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7198, 7198] 00020000:00200000:3.0:1789760796.479952:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29483008 chunk: [29483008, 29487104) 29487104, 4096 00020000:00200000:3.0:1789760796.479956:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.479971:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7198, 7198] 00000008:00010000:3.0:1789760796.479986:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.479991:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760796.479999:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29483008, 29487104) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.480011:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29483008=0x1c1e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.480062:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.480064:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29487104 00000020:00010000:3.0:1789760796.480071:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.480075:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.480078:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.480082:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.480084:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.480090:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.480096:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.480109:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760796.480256:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad180 - object 0x240000400:5 offset 29483008<>29487104 00000100:00100000:2.0:1789760796.480315:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512786304:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760796.575863:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974517 gen 3 00000008:00010000:2.0:1789760796.575937:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760796.575946:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.575955:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.575975:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000100:00100000:2.0:1789760796.575998:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512786304:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760796.576104:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7198, 7198] 00000080:00200000:3.0:1789760796.576118:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.576123:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.576134:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29487104 i_blocks 59399 00000020:00200000:3.0:1789760796.576137:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.576145:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.576170:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29487104, count: 4096 00000080:00200000:3.0:1789760796.576176:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29487104, bytes: 4096 00000080:00200000:3.0:1789760796.576184:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.576192:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.576194:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29487104, 29491200) 0 0 00000080:00200000:3.0:1789760796.576197:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.576209:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7199, 7199] 00020000:00200000:3.0:1789760796.576216:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29487104 chunk: [29487104, 29491200) 29491200, 4096 00020000:00200000:3.0:1789760796.576219:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.576234:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7199, 7199] 00000008:00010000:3.0:1789760796.576247:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.576252:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760796.576261:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29487104, 29491200) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.576271:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29487104=0x1c1f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.576314:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.576317:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29491200 00000020:00010000:3.0:1789760796.576323:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.576327:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.576330:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.576333:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.576335:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.576341:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.576348:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.576360:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760796.576436:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac380 - object 0x240000400:5 offset 29487104<>29491200 00000100:00100000:2.0:1789760796.576526:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512786432:192.168.203.156@tcp:4:dd.0 00000008:00010000:2.0:1789760796.654963:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760796.654976:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.654986:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.655020:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760796.655049:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512786432:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760796.655070:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7199, 7199] 00000080:00200000:3.0:1789760796.655085:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.655091:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.655105:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29491200 i_blocks 59399 00000020:00200000:3.0:1789760796.655109:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.655116:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.655141:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29491200, count: 4096 00000080:00200000:3.0:1789760796.655147:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29491200, bytes: 4096 00000080:00200000:3.0:1789760796.655156:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.655164:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.655166:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29491200, 29495296) 0 0 00000080:00200000:3.0:1789760796.655170:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.655183:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7200, 7200] 00020000:00200000:3.0:1789760796.655190:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29491200 chunk: [29491200, 29495296) 29495296, 4096 00020000:00200000:3.0:1789760796.655193:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.655208:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7200, 7200] 00000008:00010000:3.0:1789760796.655222:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.655227:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760796.655236:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29491200, 29495296) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.655246:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29491200=0x1c20000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.655289:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.655292:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29495296 00000020:00010000:3.0:1789760796.655298:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.655302:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.655305:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.655308:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.655310:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.655316:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.655321:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.655334:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea180 00000008:00100000:3.0:1789760796.655662:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107af480 - object 0x240000400:5 offset 29491200<>29495296 00000100:00100000:3.0:1789760796.655775:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760796.655799:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512786560:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760796.706333:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974519 gen 3 00000008:00010000:2.0:1789760796.706378:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea180 00000020:00200000:2.0:1789760796.706385:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.706393:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.706649:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea180 00000080:00200000:3.0:1789760796.706744:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7200, 7200] 00000080:00200000:3.0:1789760796.706758:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.706764:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.706776:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29495296 i_blocks 59399 00000020:00200000:3.0:1789760796.706779:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.706787:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.706811:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29495296, count: 4096 00000080:00200000:3.0:1789760796.706818:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29495296, bytes: 4096 00000080:00200000:3.0:1789760796.706827:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.706834:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.706837:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29495296, 29499392) 0 0 00000080:00200000:3.0:1789760796.706840:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.706852:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7201, 7201] 00020000:00200000:3.0:1789760796.706858:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29495296 chunk: [29495296, 29499392) 29499392, 4096 00020000:00200000:3.0:1789760796.706862:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.706876:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7201, 7201] 00000008:00010000:3.0:1789760796.706891:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.706897:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760796.706943:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29495296, 29499392) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.706954:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29495296=0x1c21000, pages 1 (max 131072), lockless, parallel 00000100:00100000:2.0:1789760796.707079:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512786560:192.168.203.156@tcp:4:dd.0 00000020:00200000:3.0:1789760796.707390:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.707394:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29499392 00000020:00010000:3.0:1789760796.707468:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.707473:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.707476:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.707480:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.707482:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.707489:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.707495:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.707511:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ead80 00000008:00100000:3.0:1789760796.707589:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ae300 - object 0x240000400:5 offset 29495296<>29499392 00000100:00100000:3.0:1789760796.707629:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760796.707642:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512786688:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760796.748414:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974520 gen 3 00000008:00010000:2.0:1789760796.748451:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ead80 00000020:00200000:2.0:1789760796.748457:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.748466:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.748485:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ead80 00000100:00100000:2.0:1789760796.748507:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512786688:192.168.203.156@tcp:4:dd.0 00000080:00200000:3.0:1789760796.748648:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7201, 7201] 00000080:00200000:3.0:1789760796.748661:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.748666:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.748677:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29499392 i_blocks 59399 00000020:00200000:3.0:1789760796.748680:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.748688:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.748712:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29499392, count: 4096 00000080:00200000:3.0:1789760796.748718:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29499392, bytes: 4096 00000080:00200000:3.0:1789760796.748724:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.748731:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.748733:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29499392, 29503488) 0 0 00000080:00200000:3.0:1789760796.748737:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.748748:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7202, 7202] 00020000:00200000:3.0:1789760796.748755:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29499392 chunk: [29499392, 29503488) 29503488, 4096 00020000:00200000:3.0:1789760796.748758:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.748772:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7202, 7202] 00000008:00010000:3.0:1789760796.748786:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.748792:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760796.748801:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29499392, 29503488) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.748812:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29499392=0x1c22000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.748853:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.748856:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29503488 00000020:00010000:3.0:1789760796.748862:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.748865:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.748868:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.748870:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.748873:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.748879:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.748884:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.748897:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039eae40 00000008:00100000:3.0:1789760796.749078:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac700 - object 0x240000400:5 offset 29499392<>29503488 00000100:00100000:2.0:1789760796.749133:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512786816:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760796.796974:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974521 gen 3 00000008:00010000:2.0:1789760796.797021:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039eae40 00000020:00200000:2.0:1789760796.797027:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.797038:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.797079:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039eae40 00000080:00200000:3.0:1789760796.797084:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7202, 7202] 00000080:00200000:3.0:1789760796.797097:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1789760796.797102:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:3.0:1789760796.797114:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29503488 i_blocks 59399 00000100:00100000:2.0:1789760796.797115:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512786816:192.168.203.156@tcp:4:dd.0 00000020:00200000:3.0:1789760796.797117:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1789760796.797124:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:3.0:1789760796.797152:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29503488, count: 4096 00000080:00200000:3.0:1789760796.797159:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29503488, bytes: 4096 00000080:00200000:3.0:1789760796.797166:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:3.0:1789760796.797173:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:3.0:1789760796.797175:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29503488, 29507584) 0 0 00000080:00200000:3.0:1789760796.797179:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:3.0:1789760796.797192:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7203, 7203] 00020000:00200000:3.0:1789760796.797198:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29503488 chunk: [29503488, 29507584) 29507584, 4096 00020000:00200000:3.0:1789760796.797202:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760796.797217:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7203, 7203] 00000008:00010000:3.0:1789760796.797230:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:3.0:1789760796.797235:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc018155130, osc lock ffff8dc010d34720, flags 0x0 00000080:00200000:3.0:1789760796.797244:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29503488, 29507584) append 0 append lockpos 0 00000080:00200000:3.0:1789760796.797255:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29503488=0x1c23000, pages 1 (max 131072), lockless, parallel 00000020:00200000:3.0:1789760796.797298:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:3.0:1789760796.797301:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29507584 00000020:00010000:3.0:1789760796.797307:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:3.0:1789760796.797311:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.797314:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc018155130 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:3.0:1789760796.797317:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:3.0:1789760796.797320:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc018155130 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:3.0:1789760796.797326:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:3.0:1789760796.797332:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760796.797346:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0039ea900 00000008:00100000:3.0:1789760796.797424:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ad500 - object 0x240000400:5 offset 29503488<>29507584 00000100:00100000:3.0:1789760796.797459:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760796.797472:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512786944:192.168.203.156@tcp:4:dd.0 00000100:00100000:2.0:1789760796.845025:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974522 gen 3 00000008:00010000:2.0:1789760796.845072:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0039ea900 00000020:00200000:2.0:1789760796.845078:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760796.845087:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760796.845173:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0039ea900 00000100:00100000:2.0:1789760796.845200:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512786944:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760796.845329:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7203, 7203] 00000080:00200000:0.0:1789760796.845343:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.845348:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.845361:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29507584 i_blocks 59399 00000020:00200000:0.0:1789760796.845364:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.845372:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.845399:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29507584, count: 4096 00000080:00200000:0.0:1789760796.845417:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29507584, bytes: 4096 00000080:00200000:0.0:1789760796.845425:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.845433:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760796.845435:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29507584, 29511680) 0 0 00000080:00200000:0.0:1789760796.845440:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.845453:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7204, 7204] 00020000:00200000:0.0:1789760796.845459:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29507584 chunk: [29507584, 29511680) 29511680, 4096 00020000:00200000:0.0:1789760796.845463:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.845478:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7204, 7204] 00000008:00010000:0.0:1789760796.845491:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.845496:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760796.845506:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29507584, 29511680) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.845516:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29507584=0x1c24000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.845559:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.845562:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29511680 00000020:00010000:0.0:1789760796.845569:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.845572:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.845575:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.845579:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.845581:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.845587:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.845593:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.845607:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000008:00100000:0.0:1789760796.845977:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514aa00 - object 0x240000400:5 offset 29507584<>29511680 00000100:00100000:1.0:1789760796.846765:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512787072:192.168.203.156@tcp:4:dd.0 00000008:00010000:1.0:1789760796.899178:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000020:00200000:1.0:1789760796.899189:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760796.899198:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760796.899219:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc9c0 00000100:00100000:1.0:1789760796.899242:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512787072:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760796.899334:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7204, 7204] 00000080:00200000:0.0:1789760796.899346:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.899352:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.899365:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29511680 i_blocks 59399 00000020:00200000:0.0:1789760796.899368:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.899375:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.899505:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29511680, count: 4096 00000080:00200000:0.0:1789760796.899513:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29511680, bytes: 4096 00000080:00200000:0.0:1789760796.899521:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.899528:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760796.899530:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29511680, 29515776) 0 0 00000080:00200000:0.0:1789760796.899534:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.899546:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7205, 7205] 00020000:00200000:0.0:1789760796.899553:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29511680 chunk: [29511680, 29515776) 29515776, 4096 00020000:00200000:0.0:1789760796.899557:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.899582:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7205, 7205] 00000008:00010000:0.0:1789760796.899596:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.899600:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760796.899610:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29511680, 29515776) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.899620:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29511680=0x1c25000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.899662:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.899665:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29515776 00000020:00010000:0.0:1789760796.899672:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.899675:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.899678:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.899693:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.899696:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.899702:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.899707:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.899719:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cccc0 00000008:00100000:0.0:1789760796.899792:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148a80 - object 0x240000400:5 offset 29511680<>29515776 00000100:00100000:1.0:1789760796.900049:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1789760796.900063:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512787200:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760796.938023:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974524 gen 3 00000008:00010000:1.0:1789760796.938070:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cccc0 00000020:00200000:1.0:1789760796.938077:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760796.938085:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760796.938105:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cccc0 00000100:00100000:1.0:1789760796.938195:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512787200:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760796.939219:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7205, 7205] 00000080:00200000:0.0:1789760796.939234:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.939239:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.939251:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29515776 i_blocks 59399 00000020:00200000:0.0:1789760796.939255:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.939264:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.939289:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29515776, count: 4096 00000080:00200000:0.0:1789760796.939296:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29515776, bytes: 4096 00000080:00200000:0.0:1789760796.939306:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.939313:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760796.939315:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29515776, 29519872) 0 0 00000080:00200000:0.0:1789760796.939319:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.939331:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7206, 7206] 00020000:00200000:0.0:1789760796.939337:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29515776 chunk: [29515776, 29519872) 29519872, 4096 00020000:00200000:0.0:1789760796.939341:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.939355:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7206, 7206] 00000008:00010000:0.0:1789760796.939367:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.939372:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760796.939381:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29515776, 29519872) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.939392:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29515776=0x1c26000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.939452:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.939454:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29519872 00000020:00010000:0.0:1789760796.939461:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.939464:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.939467:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.939470:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.939473:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.939479:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.939484:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.939504:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc900 00000008:00100000:0.0:1789760796.939584:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148700 - object 0x240000400:5 offset 29515776<>29519872 00000100:00100000:1.0:1789760796.939782:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512787328:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760796.993181:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974525 gen 3 00000008:00010000:1.0:1789760796.993217:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc900 00000020:00200000:1.0:1789760796.993223:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760796.993233:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760796.993255:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc900 00000100:00100000:1.0:1789760796.993279:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512787328:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760796.993377:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7206, 7206] 00000080:00200000:0.0:1789760796.993390:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760796.993395:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760796.993416:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29519872 i_blocks 59399 00000020:00200000:0.0:1789760796.993421:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760796.993428:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760796.993453:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29519872, count: 4096 00000080:00200000:0.0:1789760796.993460:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29519872, bytes: 4096 00000080:00200000:0.0:1789760796.993467:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760796.993474:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760796.993476:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29519872, 29523968) 0 0 00000080:00200000:0.0:1789760796.993480:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760796.993493:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7207, 7207] 00020000:00200000:0.0:1789760796.993500:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29519872 chunk: [29519872, 29523968) 29523968, 4096 00020000:00200000:0.0:1789760796.993504:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760796.993518:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7207, 7207] 00000008:00010000:0.0:1789760796.993532:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760796.993536:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760796.993545:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29519872, 29523968) append 0 append lockpos 0 00000080:00200000:0.0:1789760796.993660:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29519872=0x1c27000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760796.993704:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760796.993706:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29523968 00000020:00010000:0.0:1789760796.993713:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760796.993715:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.993718:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760796.993721:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760796.993724:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760796.993729:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760796.993734:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760796.993747:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000008:00100000:0.0:1789760796.993818:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00514ad80 - object 0x240000400:5 offset 29519872<>29523968 00000100:00100000:1.0:1789760796.993934:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512787456:192.168.203.156@tcp:4:dd.0 00000100:00100000:1.0:1789760797.065879:0:2359:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974526 gen 3 00000008:00010000:1.0:1789760797.065944:0:2359:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000020:00200000:1.0:1789760797.065951:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:1.0:1789760797.065960:0:2359:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:1.0:1789760797.066040:0:2359:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056cc6c0 00000100:00100000:1.0:1789760797.066064:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512787456:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760797.068549:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7207, 7207] 00000080:00200000:0.0:1789760797.068568:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760797.068573:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760797.068587:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29523968 i_blocks 59399 00000020:00200000:0.0:1789760797.068591:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760797.068604:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760797.068630:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29523968, count: 4096 00000080:00200000:0.0:1789760797.068638:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29523968, bytes: 4096 00000080:00200000:0.0:1789760797.068647:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760797.068655:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760797.068657:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29523968, 29528064) 0 0 00000080:00200000:0.0:1789760797.068661:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760797.068674:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7208, 7208] 00020000:00200000:0.0:1789760797.069160:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29523968 chunk: [29523968, 29528064) 29528064, 4096 00020000:00200000:0.0:1789760797.069166:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760797.069184:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7208, 7208] 00000008:00010000:0.0:1789760797.069197:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760797.069202:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760797.069211:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29523968, 29528064) append 0 append lockpos 0 00000080:00200000:0.0:1789760797.069224:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29523968=0x1c28000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760797.069264:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760797.069267:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29528064 00000020:00010000:0.0:1789760797.069274:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760797.069276:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.069279:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.069282:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760797.069284:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760797.069290:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760797.069296:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760797.069379:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccb40 00000008:00100000:0.0:1789760797.069469:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148e00 - object 0x240000400:5 offset 29523968<>29528064 00000100:00100000:0.0:1789760797.069518:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512787584:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760797.106500:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974527 gen 3 00000008:00010000:0.0:1789760797.106538:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:0.0:1789760797.106544:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.106553:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.106573:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:0.0:1789760797.106596:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512787584:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760797.106675:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7208, 7208] 00000080:00200000:2.0:1789760797.106697:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.106702:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.106713:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29528064 i_blocks 59399 00000020:00200000:2.0:1789760797.106716:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.106724:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.106747:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29528064, count: 4096 00000080:00200000:2.0:1789760797.106753:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29528064, bytes: 4096 00000080:00200000:2.0:1789760797.106760:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.106766:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760797.106768:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29528064, 29532160) 0 0 00000080:00200000:2.0:1789760797.106771:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.106783:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7209, 7209] 00020000:00200000:2.0:1789760797.106790:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29528064 chunk: [29528064, 29532160) 29532160, 4096 00020000:00200000:2.0:1789760797.106793:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.106809:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7209, 7209] 00000008:00010000:2.0:1789760797.106822:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.106827:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760797.106835:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29528064, 29532160) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.106845:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29528064=0x1c29000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.106892:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.106895:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29532160 00000020:00010000:2.0:1789760797.106933:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.106937:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.106940:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.106944:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.106946:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.106953:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.106958:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.106973:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760797.107059:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 29528064<>29532160 00000100:00100000:2.0:1789760797.107103:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512787712:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760797.138766:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974528 gen 3 00000008:00010000:3.0:1789760797.138812:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760797.138818:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.138826:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.138842:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760797.138867:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512787712:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760797.139053:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7209, 7209] 00000080:00200000:2.0:1789760797.139065:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.139070:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.139082:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29532160 i_blocks 59399 00000020:00200000:2.0:1789760797.139086:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.139094:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.139120:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29532160, count: 4096 00000080:00200000:2.0:1789760797.139126:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29532160, bytes: 4096 00000080:00200000:2.0:1789760797.139134:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.139140:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760797.139143:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29532160, 29536256) 0 0 00000080:00200000:2.0:1789760797.139146:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.139159:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7210, 7210] 00020000:00200000:2.0:1789760797.139165:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29532160 chunk: [29532160, 29536256) 29536256, 4096 00020000:00200000:2.0:1789760797.139169:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.139183:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7210, 7210] 00000008:00010000:2.0:1789760797.139197:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.139202:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760797.139211:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29532160, 29536256) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.139221:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29532160=0x1c2a000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.139263:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.139266:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29536256 00000020:00010000:2.0:1789760797.139271:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.139274:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.139277:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.139280:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.139282:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.139288:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.139294:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.139309:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d840 00000008:00100000:2.0:1789760797.139382:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f100 - object 0x240000400:5 offset 29532160<>29536256 00000100:00100000:3.0:1789760797.139494:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512787840:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760797.165079:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d840 00000020:00200000:3.0:1789760797.165092:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.165100:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.165118:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d840 00000100:00100000:3.0:1789760797.165144:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512787840:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760797.165381:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7210, 7210] 00000080:00200000:2.0:1789760797.165394:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.165410:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.165423:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29536256 i_blocks 59399 00000020:00200000:2.0:1789760797.165426:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.165435:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.165469:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29536256, count: 4096 00000080:00200000:2.0:1789760797.165476:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29536256, bytes: 4096 00000080:00200000:2.0:1789760797.165483:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.165490:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760797.165493:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29536256, 29540352) 0 0 00000080:00200000:2.0:1789760797.165497:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.165509:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7211, 7211] 00020000:00200000:2.0:1789760797.165516:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29536256 chunk: [29536256, 29540352) 29540352, 4096 00020000:00200000:2.0:1789760797.165520:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.165536:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7211, 7211] 00000008:00010000:2.0:1789760797.165550:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.165555:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760797.165566:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29536256, 29540352) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.165577:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29536256=0x1c2b000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.165623:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.165626:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29540352 00000020:00010000:2.0:1789760797.165633:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.165636:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.165639:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.165642:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.165645:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.165651:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.165657:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.165672:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d240 00000008:00100000:2.0:1789760797.165762:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368d880 - object 0x240000400:5 offset 29536256<>29540352 00000100:00100000:2.0:1789760797.165814:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512787968:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760797.196554:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974530 gen 3 00000008:00010000:3.0:1789760797.196600:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d240 00000020:00200000:3.0:1789760797.196607:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.196615:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.196634:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d240 00000100:00100000:3.0:1789760797.196659:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512787968:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760797.196746:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7211, 7211] 00000080:00200000:2.0:1789760797.196759:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.196764:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.196776:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29540352 i_blocks 59399 00000020:00200000:2.0:1789760797.196780:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.196798:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.196824:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29540352, count: 4096 00000080:00200000:2.0:1789760797.196830:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29540352, bytes: 4096 00000080:00200000:2.0:1789760797.196837:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.196843:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760797.196845:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29540352, 29544448) 0 0 00000080:00200000:2.0:1789760797.196849:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.196863:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7212, 7212] 00020000:00200000:2.0:1789760797.196870:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29540352 chunk: [29540352, 29544448) 29544448, 4096 00020000:00200000:2.0:1789760797.196873:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.196887:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7212, 7212] 00000008:00010000:2.0:1789760797.196928:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.196933:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760797.196943:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29540352, 29544448) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.196953:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29540352=0x1c2c000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.196998:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.197001:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29544448 00000020:00010000:2.0:1789760797.197018:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.197021:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.197024:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.197027:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.197029:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.197036:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.197042:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.197057:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000008:00100000:2.0:1789760797.197311:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368fb80 - object 0x240000400:5 offset 29540352<>29544448 00000100:00100000:3.0:1789760797.197373:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512788096:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760797.229813:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974531 gen 3 00000008:00010000:3.0:1789760797.229850:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000020:00200000:3.0:1789760797.229856:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.229865:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.229883:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d6c0 00000080:00200000:2.0:1789760797.230112:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7212, 7212] 00000080:00200000:2.0:1789760797.230126:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.230130:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.230141:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29544448 i_blocks 59399 00000020:00200000:2.0:1789760797.230144:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.233593:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.233632:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29544448, count: 4096 00000080:00200000:2.0:1789760797.233643:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29544448, bytes: 4096 00000080:00200000:2.0:1789760797.233656:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.233663:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760797.233666:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29544448, 29548544) 0 0 00000080:00200000:2.0:1789760797.233670:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.233686:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7213, 7213] 00020000:00200000:2.0:1789760797.233694:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29544448 chunk: [29544448, 29548544) 29548544, 4096 00020000:00200000:2.0:1789760797.233698:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.233716:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7213, 7213] 00000008:00010000:2.0:1789760797.233730:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.233734:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760797.233744:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29544448, 29548544) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.233756:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29544448=0x1c2d000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.233798:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.233801:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29548544 00000020:00010000:2.0:1789760797.233808:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.233811:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.233814:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.233817:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.233820:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.233829:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.233835:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1789760797.233942:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512788096:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760797.233983:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d300 00000008:00100000:3.0:1789760797.234074:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac000 - object 0x240000400:5 offset 29544448<>29548544 00000100:00100000:3.0:1789760797.234114:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1789760797.234126:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512788224:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760797.267966:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974532 gen 3 00000008:00010000:3.0:1789760797.268019:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d300 00000020:00200000:3.0:1789760797.268025:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.268033:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.268052:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d300 00000100:00100000:3.0:1789760797.268074:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512788224:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760797.268300:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7213, 7213] 00000080:00200000:2.0:1789760797.268314:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.268320:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.268331:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29548544 i_blocks 59399 00000020:00200000:2.0:1789760797.268334:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.268342:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.268369:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29548544, count: 4096 00000080:00200000:2.0:1789760797.268376:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29548544, bytes: 4096 00000080:00200000:2.0:1789760797.268384:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.268391:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760797.268393:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29548544, 29552640) 0 0 00000080:00200000:2.0:1789760797.268397:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.268410:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7214, 7214] 00020000:00200000:2.0:1789760797.268417:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29548544 chunk: [29548544, 29552640) 29552640, 4096 00020000:00200000:2.0:1789760797.268420:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.268437:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7214, 7214] 00000008:00010000:2.0:1789760797.268450:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.268455:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760797.268464:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29548544, 29552640) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.268474:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29548544=0x1c2e000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.268511:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.268513:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29552640 00000020:00010000:2.0:1789760797.268520:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.268523:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.268526:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.268529:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.268532:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.268539:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.268545:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:3.0:1789760797.268613:0:11838:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:3.0:1789760797.268691:0:11838:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc0107ac000 - object 0x240000400:5 offset 29548544<>29552640 00000100:00100000:3.0:1789760797.268732:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512788352:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760797.302430:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974533 gen 3 00000008:00010000:2.0:1789760797.302563:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:2.0:1789760797.302571:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:2.0:1789760797.302579:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:2.0:1789760797.302600:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:2.0:1789760797.302627:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512788352:192.168.203.156@tcp:4:dd.0 00000080:00200000:0.0:1789760797.302716:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7214, 7214] 00000080:00200000:0.0:1789760797.302729:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1789760797.302734:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:0.0:1789760797.302747:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29552640 i_blocks 59399 00000020:00200000:0.0:1789760797.302751:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:0.0:1789760797.302760:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:0.0:1789760797.302785:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29552640, count: 4096 00000080:00200000:0.0:1789760797.302792:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29552640, bytes: 4096 00000080:00200000:0.0:1789760797.302803:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:0.0:1789760797.302810:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:0.0:1789760797.302812:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29552640, 29556736) 0 0 00000080:00200000:0.0:1789760797.302815:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:0.0:1789760797.302829:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7215, 7215] 00020000:00200000:0.0:1789760797.302836:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29552640 chunk: [29552640, 29556736) 29556736, 4096 00020000:00200000:0.0:1789760797.302840:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1789760797.302865:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7215, 7215] 00000008:00010000:0.0:1789760797.302880:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:0.0:1789760797.302885:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc007c17530, osc lock ffff8dc020b43300, flags 0x0 00000080:00200000:0.0:1789760797.302895:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29552640, 29556736) append 0 append lockpos 0 00000080:00200000:0.0:1789760797.302924:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29552640=0x1c2f000, pages 1 (max 131072), lockless, parallel 00000020:00200000:0.0:1789760797.302968:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:0.0:1789760797.302971:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29556736 00000020:00010000:0.0:1789760797.302978:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:0.0:1789760797.302981:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.302985:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc007c17530 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:0.0:1789760797.302988:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:0.0:1789760797.302991:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc007c17530 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:0.0:1789760797.302997:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:0.0:1789760797.303003:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:0.0:1789760797.303027:0:11139:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc0056ccb40 00000008:00100000:0.0:1789760797.303101:0:11139:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc005148e00 - object 0x240000400:5 offset 29552640<>29556736 00000100:00100000:0.0:1789760797.303138:0:2358:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1789760797.303152:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512788480:192.168.203.156@tcp:4:dd.0 00000100:00100000:0.0:1789760797.338670:0:2358:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974534 gen 3 00000008:00010000:0.0:1789760797.338716:0:2358:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc0056ccb40 00000020:00200000:0.0:1789760797.338723:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:0.0:1789760797.338732:0:2358:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:0.0:1789760797.338797:0:2358:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc0056ccb40 00000100:00100000:0.0:1789760797.338821:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512788480:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760797.339378:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7215, 7215] 00000080:00200000:2.0:1789760797.339391:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.339461:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.339475:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29556736 i_blocks 59399 00000020:00200000:2.0:1789760797.339479:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.339487:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.339512:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29556736, count: 4096 00000080:00200000:2.0:1789760797.339518:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29556736, bytes: 4096 00000080:00200000:2.0:1789760797.339526:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.339534:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760797.339536:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29556736, 29560832) 0 0 00000080:00200000:2.0:1789760797.339540:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.339553:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7216, 7216] 00020000:00200000:2.0:1789760797.339559:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29556736 chunk: [29556736, 29560832) 29560832, 4096 00020000:00200000:2.0:1789760797.339563:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.339578:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7216, 7216] 00000008:00010000:2.0:1789760797.339592:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.339596:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760797.339605:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29556736, 29560832) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.339616:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29556736=0x1c30000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.339659:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.339662:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29560832 00000020:00010000:2.0:1789760797.339668:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.339671:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.339734:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.339739:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.339742:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.339750:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.339755:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.339774:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9db40 00000008:00100000:2.0:1789760797.340171:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368f480 - object 0x240000400:5 offset 29556736<>29560832 00000100:00100000:3.0:1789760797.340240:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760797.340461:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512788608:192.168.203.156@tcp:4:dd.0 00000008:00010000:3.0:1789760797.387704:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9db40 00000020:00200000:3.0:1789760797.387715:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.387724:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.387740:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9db40 00000100:00100000:3.0:1789760797.387765:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512788608:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760797.387863:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7216, 7216] 00000080:00200000:2.0:1789760797.387876:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.387881:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.387892:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29560832 i_blocks 59399 00000020:00200000:2.0:1789760797.387895:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.387925:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.387952:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29560832, count: 4096 00000080:00200000:2.0:1789760797.387958:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29560832, bytes: 4096 00000080:00200000:2.0:1789760797.387965:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.387973:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760797.387975:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29560832, 29564928) 0 0 00000080:00200000:2.0:1789760797.387979:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.387992:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7217, 7217] 00020000:00200000:2.0:1789760797.387999:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29560832 chunk: [29560832, 29564928) 29564928, 4096 00020000:00200000:2.0:1789760797.388003:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.388029:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7217, 7217] 00000008:00010000:2.0:1789760797.388043:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.388048:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760797.388057:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29560832, 29564928) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.388067:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29560832=0x1c31000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.388109:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.388112:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29564928 00000020:00010000:2.0:1789760797.388119:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.388122:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.388125:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.388128:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.388130:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.388136:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.388142:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.388156:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d480 00000008:00100000:2.0:1789760797.388232:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c000 - object 0x240000400:5 offset 29560832<>29564928 00000100:00100000:2.0:1789760797.388267:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1789760797.388279:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512788736:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760797.463107:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974536 gen 3 00000008:00010000:3.0:1789760797.463157:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d480 00000020:00200000:3.0:1789760797.463164:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.463183:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.463201:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d480 00000100:00100000:3.0:1789760797.463228:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512788736:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760797.463377:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7217, 7217] 00000080:00200000:2.0:1789760797.463393:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.463409:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.463421:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29564928 i_blocks 59399 00000020:00200000:2.0:1789760797.463425:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.463433:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.463462:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29564928, count: 4096 00000080:00200000:2.0:1789760797.463468:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29564928, bytes: 4096 00000080:00200000:2.0:1789760797.463478:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.463485:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760797.463487:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29564928, 29569024) 0 0 00000080:00200000:2.0:1789760797.463492:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.463506:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7218, 7218] 00020000:00200000:2.0:1789760797.463514:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29564928 chunk: [29564928, 29569024) 29569024, 4096 00020000:00200000:2.0:1789760797.463517:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.463533:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7218, 7218] 00000008:00010000:2.0:1789760797.463547:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.463552:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760797.463562:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29564928, 29569024) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.463572:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29564928=0x1c32000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.463621:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.463624:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29569024 00000020:00010000:2.0:1789760797.463630:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.463633:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.463636:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.463639:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.463642:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.463648:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.463653:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.463668:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d540 00000008:00100000:2.0:1789760797.463763:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ca80 - object 0x240000400:5 offset 29564928<>29569024 00000100:00100000:3.0:1789760797.464044:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512788864:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760797.470989:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760797.505610:0:2361:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974537 gen 3 00000008:00010000:3.0:1789760797.505650:0:2361:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9d540 00000020:00200000:3.0:1789760797.505656:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.505666:0:2361:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.505690:0:2361:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9d540 00000100:00100000:3.0:1789760797.505715:0:2361:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512788864:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760797.506078:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7218, 7218] 00000080:00200000:2.0:1789760797.506091:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.506096:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.506107:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29569024 i_blocks 59399 00000020:00200000:2.0:1789760797.506110:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.506119:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.506144:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29569024, count: 4096 00000080:00200000:2.0:1789760797.506149:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29569024, bytes: 4096 00000080:00200000:2.0:1789760797.506156:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.506162:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760797.506165:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29569024, 29573120) 0 0 00000080:00200000:2.0:1789760797.506168:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.506192:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7219, 7219] 00020000:00200000:2.0:1789760797.506199:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29569024 chunk: [29569024, 29573120) 29573120, 4096 00020000:00200000:2.0:1789760797.506203:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.506219:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7219, 7219] 00000008:00010000:2.0:1789760797.506234:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.506239:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760797.506248:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29569024, 29573120) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.506258:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29569024=0x1c33000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.506301:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.506303:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29573120 00000020:00010000:2.0:1789760797.506309:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.506313:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.506315:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.506318:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.506321:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.506326:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.506332:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.506347:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9da80 00000008:00100000:2.0:1789760797.506435:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ce00 - object 0x240000400:5 offset 29569024<>29573120 00000100:00100000:3.0:1789760797.506590:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512788992:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760797.555818:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974538 gen 3 00000008:00010000:3.0:1789760797.555856:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9da80 00000020:00200000:3.0:1789760797.555863:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.555872:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.555890:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9da80 00000100:00100000:3.0:1789760797.555941:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512788992:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760797.556074:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7219, 7219] 00000080:00200000:2.0:1789760797.556087:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.556092:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.556103:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29573120 i_blocks 59399 00000020:00200000:2.0:1789760797.556106:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.556114:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.556139:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29573120, count: 4096 00000080:00200000:2.0:1789760797.556146:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29573120, bytes: 4096 00000080:00200000:2.0:1789760797.556154:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.556160:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760797.556163:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29573120, 29577216) 0 0 00000080:00200000:2.0:1789760797.556167:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.556181:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7220, 7220] 00020000:00200000:2.0:1789760797.556187:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29573120 chunk: [29573120, 29577216) 29577216, 4096 00020000:00200000:2.0:1789760797.556191:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.556348:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7220, 7220] 00000008:00010000:2.0:1789760797.556364:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.556368:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760797.556378:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29573120, 29577216) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.556389:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29573120=0x1c34000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.556445:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.556448:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29577216 00000020:00010000:2.0:1789760797.556455:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.556459:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.556462:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.556465:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.556468:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.556474:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.556480:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.556494:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9df00 00000008:00100000:2.0:1789760797.556568:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368ea00 - object 0x240000400:5 offset 29573120<>29577216 00000100:00100000:2.0:1789760797.556609:0:2360:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1789760797.556621:0:2360:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512789120:192.168.203.156@tcp:4:dd.0 00000100:00100000:3.0:1789760797.614155:0:2360:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8dc013c56000: committing for last_committed 4294974539 gen 3 00000008:00010000:3.0:1789760797.614190:0:2360:0:(osc_cache.c:908:osc_extent_finish()) ### no dlm lock: extent: ffff8dc002d9df00 00000020:00200000:3.0:1789760797.614211:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000020:00200000:3.0:1789760797.614219:0:2360:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000008:00010000:3.0:1789760797.614232:0:2360:0:(osc_cache.c:497:__osc_extent_remove()) ### no dlm lock: extent: ffff8dc002d9df00 00000100:00100000:3.0:1789760797.614253:0:2360:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2360:1876699512789120:192.168.203.156@tcp:4:dd.0 00000080:00200000:2.0:1789760797.614333:0:11789:0:(file.c:2205:ll_file_io_generic()) Range unlock [7220, 7220] 00000080:00200000:2.0:1789760797.614346:0:11789:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1789760797.614351:0:11789:0:(file.c:2238:ll_file_io_generic()) tbf: 2 io complete with rc: 0, result: 4096, restart: 0 00000080:00200000:2.0:1789760797.614362:0:11789:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x9:0x0] updating i_size 29577216 i_blocks 59399 00000020:00200000:2.0:1789760797.614365:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1789760797.614374:0:11789:0:(file.c:2324:ll_file_io_generic()) iot: 2, result: 4096 00000080:00200008:2.0:1789760797.614409:0:11789:0:(file.c:2650:do_file_write_iter()) START file tbf:[0x200000401:0x9:0x0], ppos: 29577216, count: 4096 00000080:00200000:2.0:1789760797.614416:0:11789:0:(file.c:2090:ll_file_io_generic()) tbf: write ppos: 29577216, bytes: 4096 00000080:00200000:2.0:1789760797.614424:0:11789:0:(file.c:1848:ll_io_set_mirror()) tbf: desiginated mirror: 0 00000020:00200000:2.0:1789760797.614431:0:11789:0:(cl_io.c:214:cl_io_rw_init()) header@000000002ba0942f[0x110, 2, [0x200000401:0x9:0x0] hash] 00000020:00200000:2.0:1789760797.614433:0:11789:0:(cl_io.c:214:cl_io_rw_init()) io range: 2 [29577216, 29581312) 0 0 00000080:00200000:2.0:1789760797.614437:0:11789:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 00000080:00200000:2.0:1789760797.614451:0:11789:0:(file.c:2168:ll_file_io_generic()) Range lock [7221, 7221] 00020000:00200000:2.0:1789760797.614457:0:11789:0:(lov_io.c:1277:lov_io_rw_iter_init()) stripe: 29577216 chunk: [29577216, 29581312) 29581312, 4096 00020000:00200000:2.0:1789760797.614461:0:11789:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1789760797.614478:0:11789:0:(vvp_io.c:214:vvp_io_one_lock_index()) lock: 1 [7221, 7221] 00000008:00010000:2.0:1789760797.614491:0:11789:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: 10 00000008:00010000:2.0:1789760797.614496:0:11789:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff8dc011311630, osc lock ffff8dc003548f70, flags 0x0 00000080:00200000:2.0:1789760797.614506:0:11789:0:(vvp_io.c:1292:vvp_io_write_start()) tbf: write [29577216, 29581312) append 0 append lockpos 0 00000080:00200000:2.0:1789760797.614517:0:11789:0:(rw26.c:362:ll_direct_IO()) VFS Op:inode=[0x200000401:0x9:0x0](ffff8dc01019c2c8), size=4096 (max 536870912), offset=29577216=0x1c35000, pages 1 (max 131072), lockless, parallel 00000020:00200000:2.0:1789760797.614560:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:2.0:1789760797.614563:0:11789:0:(rw26.c:544:ll_direct_IO()) result 0 tot_bytes 4096 count 4096 file_offset 29581312 00000020:00010000:2.0:1789760797.614571:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_release():245 00000020:00010000:2.0:1789760797.614574:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.614577:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8dc011311630 (ffff8dc010db0908/1) at cl_lock_cancel():142 00000020:00010000:2.0:1789760797.614580:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc0050dcb78 (ffff8dc018e42128/0) at cl_lock_fini():72 00000020:00010000:2.0:1789760797.614582:0:11789:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8dc011311630 (ffff8dc018e42128/1) at cl_lock_fini():72 00000080:00200000:2.0:1789760797.614588:0:11789:0:(vvp_io.c:533:vvp_io_advance()) advancing 4096 bytes 00000020:00200000:2.0:1789760797.614592:0:11789:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000008:00010000:2.0:1789760797.614608:0:506:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### no dlm lock: extent: ffff8dc002d9d780 00000008:00100000:2.0:1789760797.614700:0:506:0:(osc_request.c:2000:osc_brw_prep_request()) brw rpc ffff8dc00368c380 - object 0x240000400:5 offset 29577216<>29581312 00000100:00100000:3.0:1789760797.614938:0:2361:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1789760797.614953:0:2361:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2361:1876699512789248:192.168.203.156@tcp:4:dd.0 00010000:00010000:2.0:1789760798.498687:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760799.519538:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760800.541153:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760800.541927:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e6cf7267-9b19-462a-a263-14115df15dab:2358:1876699512789376:192.168.203.156@tcp:400:kworker.0 00000100:00100000:1.0:1789760800.541960:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512789504:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1789760800.542038:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512789632:192.168.203.156@tcp:400:kworker.0 00000100:00100000:1.0:1789760800.542095:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2359:1876699512789760:192.168.203.156@tcp:400:kworker.0 00010000:00010000:2.0:1789760801.565048:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760802.589149:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760803.613160:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760804.637129:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760805.661157:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512790016:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1789760805.661288:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2359:1876699512790272:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1789760805.661336:0:2359:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:0.0:1789760805.661345:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e6cf7267-9b19-462a-a263-14115df15dab:2359:1876699512789888:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1789760805.661458:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512790144:192.168.203.156@tcp:400:kworker.0 00010000:00010000:2.0:1789760805.661565:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760806.685103:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760807.709055:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760808.733043:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760809.757030:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760810.781051:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789760810.781119:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e6cf7267-9b19-462a-a263-14115df15dab:2358:1876699512790400:192.168.203.156@tcp:400:kworker.0 00000100:00100000:1.0:1789760810.781120:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512790528:192.168.203.156@tcp:400:kworker.0 00000100:00100000:1.0:1789760810.781213:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2359:1876699512790784:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1789760810.781214:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512790656:192.168.203.156@tcp:400:kworker.0 00010000:00010000:2.0:1789760811.805068:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760812.829155:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1789760813.853039:0:2361:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760797/real 1789760797] req@ffff8dc00368c380 x1876699512789248/t0(0) o4->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:6/4 lens 4584/448 e 0 to 1 dl 1789760813 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'dd.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1789760813.853143:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760813.858794:0:2361:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1789760813.858803:0:2361:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff8dc013c56000: Connection to lustre-OST0000 (at 192.168.203.156@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1789760813.861806:0:2361:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0000_UUID: waking up pinger s:DISCONN 00000100:00080000:0.0:1789760813.862007:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760813.862014:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760813.862018:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760813.862028:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 384 00000100:00080000:0.0:1789760813.862043:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760813.862054:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760813.862077:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514bb80 x1876699512791168/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760813.862146:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512791168:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789760813.862155:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e6cf7267-9b19-462a-a263-14115df15dab:2358:1876699512790912:192.168.203.156@tcp:400:kworker.0 00000100:00100000:1.0:1789760813.862172:0:2359:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512791040:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1789760813.862270:0:2358:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2358:1876699512791296:192.168.203.156@tcp:400:kworker.0 00010000:00010000:2.0:1789760814.877057:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1789760814.877075:0:6917:0:(vvp_io.c:1826:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1789760814.877094:0:6917:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1789760814.877126:0:6917:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:2.0:1789760815.901084:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1789760816.925060:0:2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760800/real 1789760800] req@ffff8dc005c96a00 x1876699512789376/t0(0) o400->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 224/224 e 0 to 1 dl 1789760816 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:3.0:1789760816.925097:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760816.932764:0:2358:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dc005c96a00 x1876699512789376/t0(0) o400->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 224/224 e 0 to 1 dl 1789760816 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760816.932775:0:2358:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from FULL to DISCONN 00000100:02020000:1.0:1789760816.932780:0:2358:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.203.156@tcp: Connection to MGS (at 192.168.203.156@tcp) was lost; in progress operations using this service will fail 00000100:00000400:0.0:1789760816.932789:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760800/real 1789760800] req@ffff8dc005c94380 x1876699512789504/t0(0) o400->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760816 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760816.932820:0:2359:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dc005c94380 x1876699512789504/t0(0) o400->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760816 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760816.932829:0:2359:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1789760816.932833:0:2359:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff8dc013c56000: Connection to lustre-MDT0000 (at 192.168.203.156@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:1.0:1789760816.935779:0:2358:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.203.156@tcp for MGC192.168.203.156@tcp not replayable, auto-deactivating 00000100:00080000:1.0:1789760816.935781:0:2358:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:1.0:1789760816.935786:0:2358:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8dc005c96a00 x1876699512789376/t0(0) o400->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 224/224 e 0 to 1 dl 1789760816 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760816.935797:0:2358:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8dc005148e00 x1876699512789888/t0(0) o400->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 224/224 e 0 to 0 dl 1789760821 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760816.935805:0:2358:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8dc005148380 x1876699512790400/t0(0) o400->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 224/224 e 0 to 0 dl 1789760826 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760816.935811:0:2358:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8dc005148000 x1876699512790912/t0(0) o400->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 224/224 e 0 to 0 dl 1789760829 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:1.0:1789760816.935818:0:2358:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:1.0:1789760816.935820:0:2358:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:1.0:1789760816.935821:0:2358:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:1.0:1789760816.935822:0:2358:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00000400:1.0:1789760816.935839:0:2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760800/real 1789760800] req@ffff8dc005c95500 x1876699512789632/t0(0) o400->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760816 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760816.935854:0:2358:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dc005c95500 x1876699512789632/t0(0) o400->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760816 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760816.935877:0:2358:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8dc0127cd000 already not connected (conn 4, was 3): CONNECTING 00000100:00080000:3.0:1789760816.935887:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760816.935895:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1789760816.935933:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e6cf7267-9b19-462a-a263-14115df15dab:2358:1876699512790912:192.168.203.156@tcp:400:kworker.0 00000100:00080000:3.0:1789760816.935936:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789760816.935952:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e6cf7267-9b19-462a-a263-14115df15dab:2358:1876699512790400:192.168.203.156@tcp:400:kworker.0 00000100:00080000:3.0:1789760816.935954:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 240 00000100:00080000:3.0:1789760816.935981:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789760816.935990:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:e6cf7267-9b19-462a-a263-14115df15dab:2358:1876699512789376:192.168.203.156@tcp:400:kworker.0 00000100:00100000:1.0:1789760816.935996:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512789632:192.168.203.156@tcp:400:kworker.0 00000100:00100000:3.0:1789760816.936026:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512791424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760816.936087:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512791424:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789760816.939776:0:2359:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00080000:3.0:1789760816.939788:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000400:0.0:1789760816.939795:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760800/real 1789760800] req@ffff8dc005c94700 x1876699512789760/t0(0) o400->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760816 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760816.939797:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760816.939804:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760816.939818:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 240 00000100:00100000:0.0:1789760816.939823:0:2359:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dc005c94700 x1876699512789760/t0(0) o400->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760816 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760816.939832:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:0.0:1789760816.939834:0:2359:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1789760816.939842:0:2359:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff8dc003505800: Connection to lustre-MDT0000 (at 192.168.203.156@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000008:00100000:3.0:1789760816.939848:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789760816.939849:0:2359:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00100000:3.0:1789760816.939887:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699512791552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789760816.939889:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:e6cf7267-9b19-462a-a263-14115df15dab:2359:1876699512789888:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1789760816.939924:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512789504:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1789760816.939950:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2359:1876699512789760:192.168.203.156@tcp:400:kworker.0 00000100:00080000:3.0:1789760816.939959:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760816.939964:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760816.939968:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760816.939971:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760816.939973:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760816.939982:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 309 00000100:00080000:3.0:1789760816.939991:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760816.939999:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760816.940004:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512791552:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789760816.940017:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699512791680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760816.940038:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760816.940040:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1789760816.940064:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512791680:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789760817.949122:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760818.972989:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760818.972999:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760818.973003:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760818.973007:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760818.973011:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760818.973015:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760818.973019:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760818.973023:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1789760818.973030:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760819.997040:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1789760821.021029:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760805/real 1789760805] req@ffff8dc00514a300 x1876699512790144/t0(0) o400->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760821 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760821.033440:0:2359:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dc00514a300 x1876699512790144/t0(0) o400->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760821 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760821.033450:0:2359:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8dc0127cd000 already not connected (conn 4, was 3): CONNECTING 00010000:00010000:3.0:1789760821.033528:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1789760821.033534:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760805/real 1789760805] req@ffff8dc00514a680 x1876699512790016/t0(0) o400->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760821 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760821.033554:0:2359:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dc00514a680 x1876699512790016/t0(0) o400->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760821 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760821.033562:0:2359:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8dc0127ce800 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:0.0:1789760821.033567:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760805/real 1789760805] req@ffff8dc005149880 x1876699512790272/t0(0) o400->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760821 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760821.033577:0:2359:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dc005149880 x1876699512790272/t0(0) o400->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760821 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760821.033583:0:2359:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8dc003581800 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:0.0:1789760821.033594:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512790144:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1789760821.033599:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512790016:192.168.203.156@tcp:400:kworker.0 00000100:00100000:0.0:1789760821.033603:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2359:1876699512790272:192.168.203.156@tcp:400:kworker.0 00000100:00080000:1.0:1789760821.981037:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760821.981047:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760821.981052:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760821.981056:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760821.981058:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760821.981062:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760821.981065:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760821.981069:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1789760822.045066:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760823.069002:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760813/real 1789760813] req@ffff8dc00514bb80 x1876699512791168/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760823 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1789760823.069030:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760823.069032:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514bb80 x1876699512791168/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760823 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760823.069055:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760823.069069:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789760823.069072:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512791168:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1789760823.069102:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760823.069109:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760823.069112:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760823.069116:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760823.069118:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760823.069121:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760823.069124:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760823.069131:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 793 00000100:00080000:3.0:1789760823.069140:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1789760823.069146:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760823.069156:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760823.069190:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512791808/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760823.069218:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760823.069221:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1789760823.069245:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512791808:192.168.203.156@tcp:8:kworker.0 00010000:00010000:0.0:1789760824.093080:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760825.117105:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1789760826.141048:0:2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760810/real 1789760810] req@ffff8dc00514b480 x1876699512790656/t0(0) o400->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760826 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760826.141048:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760816/real 1789760816] req@ffff8dc0107adc00 x1876699512791680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760826 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1789760826.141089:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760810/real 1789760810] req@ffff8dc00514b800 x1876699512790528/t0(0) o400->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760826 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760826.141161:0:2359:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dc00514b800 x1876699512790528/t0(0) o400->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760826 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760826.141170:0:2359:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8dc0127ce800 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:2.0:1789760826.141176:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699512791680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760826 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1789760826.141183:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760810/real 1789760810] req@ffff8dc005149180 x1876699512790784/t0(0) o400->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760826 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760826.141246:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760816/real 1789760816] req@ffff8dc0107afb80 x1876699512791552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760826 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760826.141253:0:2359:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dc005149180 x1876699512790784/t0(0) o400->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760826 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760826.141290:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699512791552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760826 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760826.141291:0:2359:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8dc003581800 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:2.0:1789760826.141316:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760816/real 1789760816] req@ffff8dc0107aca80 x1876699512791424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760826 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760826.141328:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512790528:192.168.203.156@tcp:400:kworker.0 00000100:00100000:2.0:1789760826.141337:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512791424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760826 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760826.141343:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:d2c40150-e97b-40da-952b-cb9f18bf7e96:2359:1876699512790784:192.168.203.156@tcp:400:kworker.0 00000100:00080000:2.0:1789760826.141383:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00010000:00010000:0.0:1789760826.141402:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760826.141407:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789760826.141414:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512791680:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1789760826.141421:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760826.141433:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789760826.141438:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512791552:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1789760826.141443:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760826.141462:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1789760826.141464:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789760826.141466:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512791424:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789760826.141479:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760826.141491:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760826.141504:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 796 00000100:00080000:3.0:1789760826.141527:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1789760826.141538:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789760826.141629:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512791936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760826.141677:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760826.141683:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760826.141686:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789760826.141725:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512791936:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789760826.141732:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 796 00000100:00080000:3.0:1789760826.141741:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1789760826.141748:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760826.141762:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760826.141785:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512792064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760826.141829:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760826.141835:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760826.141840:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760826.141846:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760826.141848:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760826.141854:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 796 00000100:00080000:3.0:1789760826.141862:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1789760826.141933:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789760826.141937:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512792064:192.168.203.156@tcp:38:kworker.0 00000008:00100000:3.0:1789760826.141946:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760826.141993:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512792192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760826.142042:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760826.142048:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1789760826.142065:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512792192:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789760826.153036:0:2358:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dc00514b480 x1876699512790656/t0(0) o400->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 224/224 e 0 to 1 dl 1789760826 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760826.153049:0:2358:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8dc0127cd000 already not connected (conn 5, was 3): CONNECTING 00000100:00100000:1.0:1789760826.153062:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2358:1876699512790656:192.168.203.156@tcp:400:kworker.0 00010000:00010000:2.0:1789760827.165141:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760828.189020:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760828.189032:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760828.189038:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760828.189043:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760828.189046:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760828.189049:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760828.189051:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760828.189054:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1789760828.189147:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760829.213106:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1789760829.213114:0:2359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760813/real 1789760813] req@ffff8dc005149c00 x1876699512791040/t0(0) o400->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760829 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1789760829.213128:0:2358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760813/real 1789760813] req@ffff8dc005149f80 x1876699512791296/t0(0) o400->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760829 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760829.213212:0:2358:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dc005149f80 x1876699512791296/t0(0) o400->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760829 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760829.213226:0:2358:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8dc003581800 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:1.0:1789760829.213237:0:2359:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8dc005149c00 x1876699512791040/t0(0) o400->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 224/224 e 0 to 1 dl 1789760829 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760829.213246:0:2358:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:d2c40150-e97b-40da-952b-cb9f18bf7e96:2358:1876699512791296:192.168.203.156@tcp:400:kworker.0 00000100:00080000:1.0:1789760829.213276:0:2359:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8dc0127ce800 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:1.0:1789760829.213303:0:2359:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2359:1876699512791040:192.168.203.156@tcp:400:kworker.0 00010000:00010000:1.0:1789760830.237143:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760831.197111:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760831.197124:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760831.197128:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760831.197132:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760831.197136:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760831.197139:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760831.197143:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760831.197147:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1789760831.261130:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760832.285199:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.0:1789760832.477052:0:2349:0:(socklnd_cb.c:2254:ksocknal_find_timed_out_conn()) A connection with 12345-192.168.203.156@tcp (192.168.203.156:988) timed out; the network or node may be down. 00010000:00010000:1.0:1789760833.309097:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760834.333073:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760835.357115:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760836.381136:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760836.381154:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760836.381159:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760836.381164:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760836.381167:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760836.381188:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760836.381193:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760836.381201:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1789760836.381280:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760837.405150:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760838.429010:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760823/real 1789760823] req@ffff8dc0107ac000 x1876699512791808/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760838 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760838.429044:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512791808/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760838 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760838.429066:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760838.429087:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789760838.429093:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512791808:192.168.203.156@tcp:8:kworker.0 00000100:00080000:0.0:1789760838.429140:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760838.429151:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760838.429155:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760838.429159:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760838.429162:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760838.429165:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760838.429169:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760838.429176:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 802 00000100:00080000:0.0:1789760838.429188:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1789760838.429195:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760838.429206:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760838.429240:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149f80 x1876699512792320/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760838.429269:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760838.429274:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1789760838.429320:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512792320:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1789760838.429370:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760839.453415:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760840.477258:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760841.501085:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760826/real 1789760826] req@ffff8dc0107ae300 x1876699512792192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760841 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760841.501133:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512792192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760841 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760841.501147:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760826/real 1789760826] req@ffff8dc0107ac700 x1876699512792064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760841 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760841.501181:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512792064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760841 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760841.501196:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789760826/real 1789760826] req@ffff8dc0107ad500 x1876699512791936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760841 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760841.501218:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512791936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760841 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760841.501246:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760841.501270:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789760841.501277:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512792192:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1789760841.501286:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760841.501296:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789760841.501301:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512792064:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1789760841.501309:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760841.501315:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789760841.501319:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512791936:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789760841.501327:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760841.501336:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760841.501341:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760841.501348:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 805 00000100:00080000:0.0:1789760841.501426:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1789760841.501437:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789760841.501699:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149180 x1876699512792448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760841.501728:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760841.501732:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760841.501735:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760841.501741:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 805 00000100:00080000:0.0:1789760841.501746:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1789760841.501751:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760841.501765:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760841.501780:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b800 x1876699512792576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760841.501792:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760841.501796:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760841.501799:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760841.501803:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760841.501805:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789760841.501808:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512792448:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789760841.501821:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 805 00000100:00080000:0.0:1789760841.501832:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1789760841.501837:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760841.501843:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760841.501857:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149880 x1876699512792704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760841.501954:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512792576:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789760841.501966:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789760841.501991:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512792704:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789760842.525228:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760843.549328:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1789760843.549330:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760843.549337:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760843.549341:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760843.549343:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760843.549345:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760843.549348:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760843.549350:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760843.549353:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1789760844.573589:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760845.597373:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760846.557063:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760846.557075:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760846.557080:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760846.557085:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760846.557088:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760846.557093:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1789760846.557097:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760846.557100:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1789760846.621450:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760847.645302:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760848.669469:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760849.693287:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760850.717472:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760851.741511:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760851.741528:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760851.741536:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760851.741544:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760851.741551:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760851.741560:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760851.741568:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1789760851.741568:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760851.741582:0:6917:0:(pinger.c:247:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1789760852.765167:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760853.789240:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760854.813407:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760855.837235:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760856.861268:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760856.861289:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760856.861298:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760856.861306:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760856.861311:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760856.861319:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1789760856.861326:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760856.861333:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:2.0:1789760856.861637:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789760856.861675:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1022 -> 192.168.203.156:988 00000400:00000100:0.0:1789760856.861779:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789760856.861815:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789760856.861977:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789760856.862026:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1789760856.862046:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760841/real 1789760856] req@ffff8dc005149180 x1876699512792448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760856.862097:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149180 x1876699512792448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760856.862119:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000800:00000100:0.0:1789760856.862174:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00080000:1.0:1789760856.862188:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760856.862203:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512792448:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789760856.862222:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760838/real 1789760856] req@ffff8dc005149f80 x1876699512792320/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760856.862269:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149f80 x1876699512792320/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760856.862288:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760856.862308:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760856.862327:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512792320:192.168.203.156@tcp:8:kworker.0 00000100:00080000:2.0:1789760856.862371:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760856.862386:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760856.862391:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760856.862405:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 821 00000100:00080000:2.0:1789760856.862420:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1789760856.862430:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789760856.862449:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760841/real 1789760856] req@ffff8dc00514b800 x1876699512792576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760856.862471:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512792832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760856.862486:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b800 x1876699512792576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760856.862493:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760856.862522:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1789760856.862543:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760856.862544:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760856.862558:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760856.862578:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760856.862587:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 818 00000100:00080000:2.0:1789760856.862595:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1789760856.862604:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1789760856.862611:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760856.862624:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512792576:192.168.203.156@tcp:38:kworker.0 00000008:00100000:2.0:1789760856.862633:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789760856.862648:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760841/real 1789760856] req@ffff8dc005149880 x1876699512792704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760856.862668:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512792960/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760856.862692:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789760856.862694:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149880 x1876699512792704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760856.862706:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1789760856.862723:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760856.862729:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760856.862729:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760856.862734:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760856.862748:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 821 00000100:00080000:2.0:1789760856.862754:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1789760856.862759:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789760856.862765:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789760856.862767:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760856.862780:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512792704:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789760856.862800:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512793088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760856.862818:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760856.862821:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760856.862823:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789760856.862828:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512792832:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789760856.862836:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 821 00000100:00080000:2.0:1789760856.862857:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1789760856.862864:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789760856.862869:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760856.862892:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512793216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789760856.863048:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789760856.863096:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760856/real 1789760856] req@ffff8dc00368f800 x1876699512792832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760856.863114:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512792832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760856.863126:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760856.863130:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760856.863134:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512792832:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789760856.863144:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512792960:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789760856.863174:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789760856.863190:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760856/real 1789760856] req@ffff8dc00368ed80 x1876699512792960/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760856.863204:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512792960/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760856.863214:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760856.863218:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760856.863222:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512792960:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789760856.863242:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512793088:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789760856.863264:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789760856.863286:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760856/real 1789760856] req@ffff8dc00368e300 x1876699512793088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760856.863301:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512793088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760856.863311:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760856.863315:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760856.863319:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512793088:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789760856.863326:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512793216:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789760856.863344:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789760856.863363:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760856/real 1789760856] req@ffff8dc00368df80 x1876699512793216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760856.863401:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512793216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760856.863418:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760856.863425:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760856.863432:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512793216:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789760857.885812:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760858.909103:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760859.933359:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760860.957102:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760861.981105:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760861.981325:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760861.981334:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760861.981339:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760861.981350:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 836 00000100:00080000:3.0:1789760861.981360:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760861.981367:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789760861.981412:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512793344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760861.981439:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760861.981442:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760861.981444:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760861.981447:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 836 00000100:00080000:3.0:1789760861.981449:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760861.981452:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760861.981461:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760861.981474:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512793472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760861.981483:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512793344:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789760861.981491:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760861.981496:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760861.981498:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760861.981503:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 836 00000100:00080000:3.0:1789760861.981507:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760861.981511:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760861.981518:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760861.981558:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512793600/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760861.981574:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760861.981578:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760861.981580:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760861.981585:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 836 00000100:00080000:3.0:1789760861.981590:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760861.981596:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760861.981603:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760861.981620:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512793728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760861.981621:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512793472:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789760861.981665:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512793600:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789760861.981696:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512793728:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789760863.005770:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789760863.517275:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1022 -> 192.168.203.156:988 00000400:00000100:0.0:1789760863.517330:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789760863.517350:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789760863.517394:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789760863.517402:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789760863.517409:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1789760863.517581:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760861/real 1789760863] req@ffff8dc0107aed80 x1876699512793728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760863.517640:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512793728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760863.517659:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760863.517667:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760863.517683:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512793728:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789760863.517697:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760861/real 1789760863] req@ffff8dc0107ac380 x1876699512793472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760863.517751:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512793472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760863.517766:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760863.517772:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760863.517779:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512793472:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789760863.517792:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760861/real 1789760863] req@ffff8dc0107ad180 x1876699512793600/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760863.517820:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512793600/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760863.517833:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760863.517839:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760863.517845:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512793600:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789760863.517857:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760861/real 1789760863] req@ffff8dc0107af480 x1876699512793344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760863.517878:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512793344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760863.517892:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760863.517897:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760863.517986:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512793344:192.168.203.156@tcp:250:kworker.0 00010000:00010000:3.0:1789760864.029449:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760865.053399:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760866.077702:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760867.101056:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760867.101075:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760867.101082:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760867.101102:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 841 00000100:00080000:3.0:1789760867.101121:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760867.101135:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789760867.101209:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512793856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760867.101262:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760867.101269:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760867.101272:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760867.101279:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 841 00000100:00080000:3.0:1789760867.101286:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760867.101294:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760867.101310:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760867.101333:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512793856:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789760867.101336:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512793984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760867.101372:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760867.101377:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760867.101380:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760867.101387:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 841 00000100:00080000:3.0:1789760867.101399:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760867.101405:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760867.101415:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00010000:00010000:2.0:1789760867.101437:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760867.101439:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512794112/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760867.101459:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760867.101465:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760867.101469:0:38:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760867.101477:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 841 00000100:00080000:3.0:1789760867.101501:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760867.101510:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760867.101518:0:38:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760867.101537:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512794240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760867.101563:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512793984:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789760867.101629:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512794112:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789760867.101685:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512794240:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789760868.125154:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760869.149337:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789760870.174120:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1022 -> 192.168.203.156:988 00000400:00000100:2.0:1789760870.174221:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00010000:00010000:0.0:1789760870.174234:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.0:1789760870.174257:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789760870.174337:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789760870.174372:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789760870.174393:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:0.0:1789760870.174553:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760867/real 1789760870] req@ffff8dc0107adf80 x1876699512793984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760870.174601:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512793984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760870.174623:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760870.174633:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789760870.174641:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512793984:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789760870.174656:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760867/real 1789760870] req@ffff8dc0107af800 x1876699512794112/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760870.174683:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512794112/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760870.174700:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760870.174707:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789760870.174714:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512794112:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789760870.174729:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760867/real 1789760870] req@ffff8dc0107ae680 x1876699512794240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760870.174768:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512794240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760870.174783:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760870.174791:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789760870.174797:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512794240:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789760870.174811:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760867/real 1789760870] req@ffff8dc0107aea00 x1876699512793856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789760870.174836:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512793856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789760870.174864:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789760870.174870:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789760870.174876:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512793856:192.168.203.156@tcp:250:kworker.0 00010000:00010000:0.0:1789760871.197275:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760872.221194:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789760872.221237:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760872.221244:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789760872.221247:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760872.221257:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 846 00000100:00080000:2.0:1789760872.221268:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760872.221274:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789760872.221307:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512794368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760872.221339:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760872.221343:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760872.221344:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760872.221348:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 846 00000100:00080000:2.0:1789760872.221352:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760872.221356:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789760872.221366:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760872.221393:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512794496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760872.221402:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760872.221406:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789760872.221407:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760872.221411:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 846 00000100:00080000:2.0:1789760872.221415:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760872.221418:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789760872.221422:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789760872.221431:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512794624/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789760872.221440:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789760872.221443:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789760872.221444:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789760872.221466:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 846 00000100:00080000:2.0:1789760872.221470:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789760872.221474:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789760872.221478:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760872.221488:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512794752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760872.221613:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512794368:192.168.203.156@tcp:250:kworker.0 00000800:00000100:3.0:1789760872.221678:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760872.221780:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760872/real 1789760872] req@ffff8dc00368e680 x1876699512794368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760872.221801:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512794368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760872.221813:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760872.221817:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760872.221822:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512794368:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789760872.221830:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512794496:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760872.221855:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760872.221869:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760872/real 1789760872] req@ffff8dc00368d180 x1876699512794496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760872.221883:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512794496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760872.221892:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760872.221895:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760872.221898:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512794496:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789760872.221935:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512794624:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789760872.221962:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760872.221975:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760872/real 1789760872] req@ffff8dc00368c700 x1876699512794624/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760872.221988:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512794624/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760872.222003:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760872.222007:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760872.222011:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512794624:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789760872.222017:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512794752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760872.222038:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760872.222056:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760872/real 1789760872] req@ffff8dc00368dc00 x1876699512794752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760872.222069:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512794752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760872.222077:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760872.222081:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760872.222084:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512794752:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789760873.245142:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760874.269261:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760875.293339:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760876.317118:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760877.341135:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760877.341153:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789760877.341161:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760877.341177:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 851 00000100:00080000:1.0:1789760877.341194:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760877.341208:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789760877.341267:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512794880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760877.341324:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760877.341335:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760877.341340:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789760877.341380:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512794880:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789760877.341384:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 851 00000100:00080000:1.0:1789760877.341401:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760877.341434:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789760877.341451:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760877.341485:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512795008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760877.341529:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760877.341537:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789760877.341542:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760877.341551:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 851 00000100:00080000:1.0:1789760877.341561:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760877.341575:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789760877.341577:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512795008:192.168.203.156@tcp:38:kworker.0 00000008:00100000:1.0:1789760877.341589:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789760877.341624:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512795136/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760877.341650:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760877.341657:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760877.341662:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760877.341670:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 851 00000100:00080000:1.0:1789760877.341680:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1789760877.341687:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760877.341690:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789760877.341705:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760877.341731:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512795264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760877.341771:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512795136:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789760877.342352:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512795264:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789760878.365855:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760879.389107:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760880.413116:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789760880.413493:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1022 -> 192.168.203.156:988 00000400:00000100:2.0:1789760880.413535:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789760880.413549:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789760880.413582:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789760880.413590:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789760880.413599:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:3.0:1789760880.413617:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760877/real 1789760880] req@ffff8dc005c96300 x1876699512795136/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760880.413649:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512795136/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760880.413663:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760880.413669:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760880.413674:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512795136:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789760880.413683:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760877/real 1789760880] req@ffff8dc005c96a00 x1876699512795008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760880.413700:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512795008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760880.413717:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760880.413722:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760880.413726:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512795008:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789760880.413734:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760877/real 1789760880] req@ffff8dc005c95500 x1876699512794880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760880.413744:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512794880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760880.413751:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760880.413753:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760880.413756:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512794880:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789760880.413778:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760877/real 1789760880] req@ffff8dc005c95f80 x1876699512795264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760880.413794:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512795264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760880.413803:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760880.413807:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760880.413811:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512795264:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789760881.437288:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760882.461243:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760882.462363:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760882.462380:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760882.462383:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760882.462391:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 857 00000100:00080000:0.0:1789760882.462400:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760882.462405:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789760882.462436:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514a680 x1876699512795392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760882.462458:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760882.462463:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760882.462464:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760882.462469:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 857 00000100:00080000:0.0:1789760882.462472:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760882.462478:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760882.462487:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760882.462509:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512795392:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789760882.462519:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514a300 x1876699512795520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760882.462532:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760882.462535:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760882.462537:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760882.462540:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 857 00000100:00080000:0.0:1789760882.462545:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760882.462548:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760882.462552:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760882.462563:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148e00 x1876699512795648/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760882.462573:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760882.462576:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760882.462578:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760882.462581:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 857 00000100:00080000:0.0:1789760882.462584:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760882.462587:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760882.462592:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760882.462599:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149500 x1876699512795776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789760882.462607:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760882.462656:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760882/real 1789760882] req@ffff8dc00514a680 x1876699512795392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760882.462672:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514a680 x1876699512795392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760882.462682:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760882.462686:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760882.462689:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512795392:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789760882.462715:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512795520:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760882.462738:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760882.462752:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760882/real 1789760882] req@ffff8dc00514a300 x1876699512795520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760882.462784:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514a300 x1876699512795520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760882.462796:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760882.462798:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760882.462801:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512795520:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789760882.462806:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512795648:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789760882.462824:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760882.462841:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760882/real 1789760882] req@ffff8dc005148e00 x1876699512795648/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760882.462850:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148e00 x1876699512795648/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760882.462855:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760882.462858:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760882.462860:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512795648:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789760882.462864:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512795776:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760882.462877:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760882.462884:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760882/real 1789760882] req@ffff8dc005149500 x1876699512795776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760882.462891:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149500 x1876699512795776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760882.462896:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760882.462898:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760882.462923:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512795776:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789760883.485309:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760884.509404:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760885.533171:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760886.557062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760887.581092:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760887.581118:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760887.581129:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789760887.581136:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760887.581149:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 862 00000100:00080000:1.0:1789760887.581162:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760887.581170:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789760887.581219:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512795904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760887.581244:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760887.581247:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760887.581249:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760887.581252:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 862 00000100:00080000:1.0:1789760887.581255:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760887.581259:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789760887.581269:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760887.581280:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512795904:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789760887.581287:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512796032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760887.581300:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760887.581304:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789760887.581306:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760887.581311:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 862 00000100:00080000:1.0:1789760887.581315:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760887.581320:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789760887.581368:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1789760887.581379:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789760887.581382:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512796160/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760887.581398:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760887.581403:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760887.581405:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760887.581410:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 862 00000100:00080000:1.0:1789760887.581415:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760887.581420:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789760887.581426:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760887.581435:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760887/real 1789760887] req@ffff8dc005c95c00 x1876699512795904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760887.581442:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512796288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760887.581464:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512795904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760887.581475:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760887.581479:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760887.581483:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512795904:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789760887.581510:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512796032:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760887.581536:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760887.581551:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760887/real 1789760887] req@ffff8dc005c97800 x1876699512796032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760887.581564:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512796032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760887.581570:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760887.581573:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760887.581575:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512796032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789760887.581580:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512796160:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789760887.581597:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760887.581605:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760887/real 1789760887] req@ffff8dc005c94a80 x1876699512796160/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760887.581613:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512796160/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760887.581619:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760887.581621:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760887.581623:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512796160:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789760887.581627:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512796288:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760887.581638:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760887.581645:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760887/real 1789760887] req@ffff8dc005c95180 x1876699512796288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760887.581660:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512796288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760887.581666:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760887.581669:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760887.581671:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512796288:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789760888.605117:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760889.629481:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760890.653275:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760891.677100:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760892.701014:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760892.701032:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760892.701039:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760892.701060:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 867 00000100:00080000:3.0:1789760892.701075:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760892.701087:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789760892.701140:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512796416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760892.701180:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760892.701185:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760892.701187:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760892.701192:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 867 00000100:00080000:3.0:1789760892.701197:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760892.701201:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760892.701215:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760892.701230:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512796544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760892.701240:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760892.701243:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760892.701245:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760892.701249:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 867 00000100:00080000:3.0:1789760892.701254:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760892.701257:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760892.701262:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760892.701274:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512796672/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760892.701290:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760892.701307:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760892.701309:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760892.701314:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 867 00000100:00080000:3.0:1789760892.701318:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760892.701323:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760892.701328:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760892.701341:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512796800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760892.701397:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512796416:192.168.203.156@tcp:250:kworker.0 00010000:00010000:2.0:1789760892.701506:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760892.701531:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512796544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789760892.701576:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512796672:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789760892.701605:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512796800:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789760893.725164:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760894.749161:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789760894.749543:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1022 -> 192.168.203.156:988 00000400:00000100:3.0:1789760894.749600:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789760894.749623:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789760894.749667:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789760894.749679:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789760894.749690:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1789760894.749717:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760892/real 1789760894] req@ffff8dc0107af100 x1876699512796416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760894.749752:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512796416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760894.749775:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760894.749781:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760894.749786:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512796416:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789760894.749796:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760892/real 1789760894] req@ffff8dc0107ace00 x1876699512796544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760894.749812:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512796544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760894.749823:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760894.749827:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760894.749830:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512796544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789760894.749838:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760892/real 1789760894] req@ffff8dc0107ad880 x1876699512796672/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760894.749852:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512796672/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760894.749860:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760894.749864:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760894.749868:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512796672:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789760894.749876:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760892/real 1789760894] req@ffff8dc0107aea00 x1876699512796800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760894.749889:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512796800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760894.749897:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760894.749941:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760894.749945:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512796800:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789760895.773420:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760896.797472:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760897.821217:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:1.0:1789760897.821225:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760897.821230:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760897.821235:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760897.821245:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 872 00000100:00080000:3.0:1789760897.821272:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760897.821280:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789760897.821318:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512796928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760897.821360:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760897.821364:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760897.821365:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760897.821370:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 872 00000100:00080000:3.0:1789760897.821374:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760897.821377:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760897.821387:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760897.821402:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512797056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760897.821411:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512796928:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789760897.821417:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760897.821427:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760897.821429:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760897.821435:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 872 00000100:00080000:3.0:1789760897.821440:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760897.821444:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760897.821450:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760897.821465:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512797184/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789760897.821474:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789760897.821485:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760897.821489:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760897.821491:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760897.821501:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 872 00000100:00080000:3.0:1789760897.821507:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789760897.821508:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760897/real 1789760897] req@ffff8dc0107ae680 x1876699512796928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760897.821516:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760897.821526:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760897.821533:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512796928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760897.821543:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789760897.821544:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512797312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760897.821548:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760897.821556:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512796928:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789760897.821591:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512797056:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789760897.821607:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789760897.821616:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760897/real 1789760897] req@ffff8dc0107af800 x1876699512797056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760897.821624:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512797056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760897.821630:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760897.821632:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760897.821635:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512797056:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789760897.821639:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512797184:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789760897.821650:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789760897.821656:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760897/real 1789760897] req@ffff8dc0107adf80 x1876699512797184/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760897.821664:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512797184/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760897.821669:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760897.821671:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760897.821673:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512797184:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789760897.821676:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512797312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789760897.821686:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789760897.821692:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760897/real 1789760897] req@ffff8dc0107af480 x1876699512797312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760897.821699:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512797312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760897.821704:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760897.821706:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760897.821708:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512797312:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789760898.845058:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760899.869194:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760900.893418:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760901.917326:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760902.941065:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760902.941166:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760902.941177:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760902.941182:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760902.941196:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 877 00000100:00080000:3.0:1789760902.941212:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760902.941231:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789760902.941301:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512797440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760902.941339:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760902.941343:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760902.941346:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760902.941350:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 877 00000100:00080000:3.0:1789760902.941354:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760902.941360:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760902.941387:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760902.941404:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512797440:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789760902.941405:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512797568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760902.941423:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760902.941430:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760902.941432:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760902.941437:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 877 00000100:00080000:3.0:1789760902.941441:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760902.941445:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760902.941451:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760902.941464:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512797696/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760902.941475:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760902.941478:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760902.941480:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760902.941484:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 877 00000100:00080000:3.0:1789760902.941488:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760902.941492:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1789760902.941497:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789760902.941499:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760902.941513:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512797824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760902.941562:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760902/real 1789760902] req@ffff8dc0107ad180 x1876699512797440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760902.941580:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512797440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760902.941591:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760902.941595:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789760902.941599:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512797440:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789760902.941626:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512797568:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789760902.941643:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789760902.941659:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760902/real 1789760902] req@ffff8dc0107ac380 x1876699512797568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760902.941682:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512797568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760902.941692:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760902.941696:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789760902.941700:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512797568:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789760902.941707:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512797696:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789760902.941729:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789760902.941738:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760902/real 1789760902] req@ffff8dc0107aed80 x1876699512797696/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760902.941747:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512797696/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760902.941762:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760902.941765:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789760902.941767:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512797696:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789760902.941771:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512797824:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789760902.941786:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789760902.941793:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760902/real 1789760902] req@ffff8dc0107ad500 x1876699512797824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760902.941801:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512797824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760902.941806:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760902.941808:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789760902.941810:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512797824:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789760903.965268:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760904.989577:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760906.013299:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760907.037170:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760908.061240:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760908.061537:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760908.061549:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789760908.061555:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760908.061570:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 882 00000100:00080000:1.0:1789760908.061583:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760908.061594:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789760908.061658:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512797952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760908.061705:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760908.061712:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760908.061716:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760908.061722:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 882 00000100:00080000:1.0:1789760908.061730:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760908.061737:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789760908.061751:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760908.061776:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512798080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760908.061795:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760908.061800:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789760908.061803:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760908.061809:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 882 00000100:00080000:1.0:1789760908.061816:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760908.061823:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789760908.061831:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789760908.061850:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512798208/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760908.061867:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512797952:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789760908.061868:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760908.061874:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760908.061877:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760908.061882:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 882 00000100:00080000:1.0:1789760908.061888:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760908.061893:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1789760908.061986:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789760908.062040:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760908/real 1789760908] req@ffff8dc005c97100 x1876699512797952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789760908.062046:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760908.062068:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512798336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760908.062083:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512797952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760908.062098:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760908.062106:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789760908.062112:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512797952:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789760908.062141:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512798080:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789760908.062202:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789760908.062225:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760908/real 1789760908] req@ffff8dc005c95180 x1876699512798080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760908.062246:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512798080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760908.062259:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760908.062263:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789760908.062267:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512798080:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789760908.062292:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512798208:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789760908.062318:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789760908.062330:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760908/real 1789760908] req@ffff8dc005c94a80 x1876699512798208/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760908.062343:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512798208/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760908.062359:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760908.062362:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789760908.062365:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512798208:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789760908.062371:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512798336:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789760908.062388:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789760908.062398:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760908/real 1789760908] req@ffff8dc005c97800 x1876699512798336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760908.062410:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512798336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760908.062418:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760908.062421:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789760908.062424:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512798336:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789760909.085125:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760910.109382:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760911.133074:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760912.157112:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760913.181123:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760913.181152:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760913.181158:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789760913.181174:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760913.181175:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 887 00000100:00080000:3.0:1789760913.181203:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760913.181216:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789760913.181271:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512798464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760913.181312:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760913.181318:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760913.181327:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760913.181336:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 887 00000100:00080000:3.0:1789760913.181342:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760913.181348:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760913.181363:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760913.181393:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512798592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760913.181408:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760913.181414:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760913.181418:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789760913.181425:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512798464:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789760913.181427:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 887 00000100:00080000:3.0:1789760913.181437:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760913.181446:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760913.181462:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760913.181485:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512798720/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760913.181558:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512798592:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1789760913.181570:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760913.181576:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760913.181579:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760913.181586:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 887 00000100:00080000:3.0:1789760913.181592:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760913.181598:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760913.181605:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760913.181607:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512798720:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789760913.181624:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512798848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760913.181655:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512798848:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789760914.205326:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760915.229384:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789760916.253266:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00010000:00010000:0.0:1789760916.253312:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789760916.253330:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789760916.253353:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789760916.253405:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789760916.253416:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789760916.253425:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1789760916.253531:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760913/real 1789760916] req@ffff8dc0107aca80 x1876699512798848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760916.253571:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512798848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760916.253586:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760916.253593:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789760916.253598:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512798848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789760916.253609:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760913/real 1789760916] req@ffff8dc0107ac000 x1876699512798720/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760916.253624:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512798720/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760916.253633:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760916.253637:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789760916.253640:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512798720:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789760916.253648:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760913/real 1789760916] req@ffff8dc0107ac700 x1876699512798464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760916.253663:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512798464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760916.253673:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760916.253676:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789760916.253680:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512798464:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789760916.253689:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760913/real 1789760916] req@ffff8dc0107ae300 x1876699512798592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789760916.253701:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512798592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789760916.253710:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789760916.253714:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789760916.253718:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512798592:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789760917.277201:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760918.301116:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760918.301131:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760918.301137:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760918.301164:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 892 00000100:00080000:3.0:1789760918.301179:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760918.301189:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789760918.301253:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699512798976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760918.301293:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1789760918.301295:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760918.301300:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760918.301304:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760918.301309:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 892 00000100:00080000:3.0:1789760918.301314:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760918.301320:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760918.301333:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760918.301349:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699512799104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760918.301363:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512798976:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789760918.301368:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760918.301380:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760918.301383:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760918.301389:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 892 00000100:00080000:3.0:1789760918.301394:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760918.301399:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760918.301406:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760918.301425:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512799232/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760918.301439:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760918.301444:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760918.301446:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760918.301450:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 892 00000100:00080000:3.0:1789760918.301456:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789760918.301460:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789760918.301467:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760918.301477:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760918.301494:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512799360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760918.301519:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760918/real 1789760918] req@ffff8dc0107afb80 x1876699512798976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760918.301550:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699512798976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760918.301563:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760918.301568:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760918.301572:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512798976:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789760918.301599:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512799104:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789760918.301625:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789760918.301644:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760918/real 1789760918] req@ffff8dc0107adc00 x1876699512799104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760918.301657:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699512799104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760918.301665:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760918.301668:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760918.301677:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512799104:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789760918.301686:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512799232:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789760918.301706:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789760918.301716:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760918/real 1789760918] req@ffff8dc0107ae300 x1876699512799232/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760918.301728:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512799232/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760918.301741:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760918.301744:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760918.301748:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512799232:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789760918.301754:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512799360:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789760918.301770:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789760918.301781:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760918/real 1789760918] req@ffff8dc0107ac700 x1876699512799360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760918.301791:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512799360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760918.301799:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760918.301802:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760918.301805:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512799360:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789760919.325289:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760920.349204:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760921.373255:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760922.397345:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760923.420996:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760923.421033:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760923.421046:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760923.421083:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 898 00000100:00080000:3.0:1789760923.421103:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760923.421116:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789760923.421159:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760923.421183:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512799488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760923.421219:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760923.421226:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760923.421230:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760923.421238:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 898 00000100:00080000:3.0:1789760923.421245:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760923.421252:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789760923.421263:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512799488:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789760923.421269:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760923.421308:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512799616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760923.421338:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760923.421344:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760923.421348:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760923.421355:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 898 00000100:00080000:3.0:1789760923.421362:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789760923.421365:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789760923.421378:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760923.421391:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760923.421420:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512799744/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760923.421435:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789760923.421442:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760923/real 1789760923] req@ffff8dc0107ac000 x1876699512799488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760923.421444:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760923.421451:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789760923.421470:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512799488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760923.421482:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 898 00000100:00080000:1.0:1789760923.421486:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760923.421508:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760923.421513:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760923.421533:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512799488:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789760923.421534:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760923.421552:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760923.421587:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512799872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760923.421594:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512799616:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789760923.421651:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789760923.421676:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760923/real 1789760923] req@ffff8dc0107aca80 x1876699512799616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760923.421697:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512799616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760923.421710:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760923.421716:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760923.421721:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512799616:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789760923.421732:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512799744:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789760923.421771:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789760923.421783:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760923/real 1789760923] req@ffff8dc0107ad500 x1876699512799744/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760923.421802:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512799744/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760923.421810:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760923.421813:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760923.421817:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512799744:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789760923.421832:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512799872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789760923.421849:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789760923.421860:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760923/real 1789760923] req@ffff8dc0107aed80 x1876699512799872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760923.421875:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512799872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760923.421901:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760923.421904:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760923.421908:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512799872:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789760924.445390:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760925.470012:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760926.493136:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760927.517609:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760928.541164:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760928.541187:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760928.541198:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760928.541204:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760928.541223:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 903 00000100:00080000:3.0:1789760928.541237:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760928.541247:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789760928.541309:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512800000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760928.541340:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760928.541344:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760928.541346:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760928.541352:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 903 00000100:00080000:3.0:1789760928.541357:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760928.541363:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789760928.541374:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512800000:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789760928.541376:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760928.541397:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512800128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760928.541407:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760928.541410:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760928.541411:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760928.541415:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 903 00000100:00080000:3.0:1789760928.541418:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760928.541421:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760928.541425:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760928.541433:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512800256/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760928.541441:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760928.541444:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760928.541445:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760928.541448:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 903 00000100:00080000:3.0:1789760928.541457:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760928.541460:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760928.541463:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789760928.541469:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760928.541473:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512800384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760928.541516:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760928/real 1789760928] req@ffff8dc0107ac380 x1876699512800000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760928.541543:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512800000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760928.541552:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760928.541556:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760928.541559:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512800000:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789760928.541581:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512800128:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789760928.541599:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789760928.541609:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760928/real 1789760928] req@ffff8dc0107ad180 x1876699512800128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760928.541617:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512800128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760928.541628:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760928.541630:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760928.541633:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512800128:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789760928.541636:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512800256:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789760928.541649:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789760928.541655:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760928/real 1789760928] req@ffff8dc0107af480 x1876699512800256/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760928.541670:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512800256/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760928.541676:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760928.541678:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760928.541680:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512800256:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789760928.541684:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512800384:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789760928.541696:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789760928.541702:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760928/real 1789760928] req@ffff8dc0107adf80 x1876699512800384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760928.541710:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512800384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760928.541715:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760928.541717:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760928.541721:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512800384:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789760929.565298:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760930.589271:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760931.613164:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760932.637242:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760933.661019:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760933.661031:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789760933.661037:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760933.661050:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 908 00000100:00080000:3.0:1789760933.661060:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760933.661069:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789760933.661099:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760933.661119:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512800512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760933.661153:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760933.661156:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760933.661158:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760933.661163:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 908 00000100:00080000:3.0:1789760933.661167:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760933.661172:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760933.661184:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760933.661193:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512800512:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789760933.661200:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512800640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789760933.661214:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760933.661218:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789760933.661220:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760933.661224:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 908 00000100:00080000:3.0:1789760933.661228:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760933.661232:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760933.661238:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789760933.661251:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512800768/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789760933.661271:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789760933.661280:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789760933.661285:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789760933.661287:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789760933.661293:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 908 00000100:00080000:3.0:1789760933.661297:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789760933.661302:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789760933.661308:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760933.661311:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760933/real 1789760933] req@ffff8dc0107af800 x1876699512800512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760933.661324:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512800896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789760933.661335:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512800512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760933.661353:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760933.661358:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760933.661363:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512800512:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789760933.661385:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512800640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789760933.661406:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789760933.661418:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760933/real 1789760933] req@ffff8dc0107ae680 x1876699512800640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760933.661431:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512800640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760933.661441:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760933.661444:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760933.661448:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512800640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789760933.661455:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512800768:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789760933.661472:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789760933.661482:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760933/real 1789760933] req@ffff8dc0107aea00 x1876699512800768/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760933.661502:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512800768/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760933.661511:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760933.661515:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760933.661518:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512800768:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789760933.661525:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512800896:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789760933.661542:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789760933.661553:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760933/real 1789760933] req@ffff8dc0107ad880 x1876699512800896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760933.661565:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512800896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789760933.661574:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789760933.661577:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789760933.661581:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512800896:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789760934.685364:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760935.709660:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760936.733191:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760937.757134:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760938.781488:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760938.781563:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760938.781572:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789760938.781577:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760938.781590:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 913 00000100:00080000:1.0:1789760938.781613:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760938.781622:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789760938.781687:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512801024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760938.781734:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760938.781739:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760938.781741:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760938.781746:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 913 00000100:00080000:1.0:1789760938.781750:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760938.781755:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789760938.781768:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760938.781785:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512801152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760938.781797:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760938.781801:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789760938.781803:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760938.781808:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 913 00000100:00080000:1.0:1789760938.781811:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760938.781814:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789760938.781819:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789760938.781829:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512801280/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760938.781837:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760938.781839:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760938.781841:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760938.781845:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 913 00000100:00080000:1.0:1789760938.781848:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760938.781852:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789760938.781855:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760938.781862:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512801408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760938.781875:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512801024:192.168.203.156@tcp:250:kworker.0 00000800:00000100:3.0:1789760938.781986:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760938.782025:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760938/real 1789760938] req@ffff8dc005c95c00 x1876699512801024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760938.782045:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512801024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760938.782058:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760938.782062:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760938.782066:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512801024:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789760938.782074:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512801152:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760938.782098:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760938.782111:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760938/real 1789760938] req@ffff8dc005c95f80 x1876699512801152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760938.782124:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512801152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760938.782133:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760938.782137:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760938.782141:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512801152:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789760938.782147:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512801280:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789760938.782168:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760938.782179:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760938/real 1789760938] req@ffff8dc005c95500 x1876699512801280/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760938.782192:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512801280/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760938.782201:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760938.782205:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760938.782208:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512801280:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789760938.782232:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512801408:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760938.782252:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760938.782266:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760938/real 1789760938] req@ffff8dc005c96a00 x1876699512801408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760938.782279:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512801408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760938.782288:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760938.782292:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760938.782295:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512801408:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789760939.805175:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760940.829257:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760941.853564:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760942.877351:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760943.901347:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760943.901639:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760943.901647:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760943.901651:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760943.901660:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 918 00000100:00080000:0.0:1789760943.901678:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760943.901685:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789760943.901720:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b100 x1876699512801536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760943.901747:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760943.901750:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760943.901753:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760943.901756:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 918 00000100:00080000:0.0:1789760943.901759:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760943.901762:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760943.901777:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760943.901792:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512801536:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789760943.901793:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149500 x1876699512801664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760943.901810:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760943.901814:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760943.901816:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760943.901828:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 918 00000100:00080000:0.0:1789760943.901833:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760943.901838:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760943.901844:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760943.901858:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148e00 x1876699512801792/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760943.901869:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760943.901872:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760943.901876:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789760943.901877:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1789760943.901883:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 918 00000100:00080000:0.0:1789760943.901889:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760943.901894:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760943.901934:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760943.901950:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514a300 x1876699512801920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760943.901971:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760943/real 1789760943] req@ffff8dc00514b100 x1876699512801536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760943.901992:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b100 x1876699512801536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760943.902004:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760943.902008:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760943.902013:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512801536:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789760943.902037:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512801664:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760943.902064:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760943.902078:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760943/real 1789760943] req@ffff8dc005149500 x1876699512801664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760943.902089:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149500 x1876699512801664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760943.902095:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760943.902097:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760943.902100:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512801664:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789760943.902105:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512801792:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789760943.902123:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760943.902130:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760943/real 1789760943] req@ffff8dc005148e00 x1876699512801792/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760943.902138:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148e00 x1876699512801792/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760943.902144:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760943.902146:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760943.902148:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512801792:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789760943.902152:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512801920:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760943.902162:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760943.902168:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760943/real 1789760943] req@ffff8dc00514a300 x1876699512801920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760943.902175:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514a300 x1876699512801920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760943.902180:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760943.902182:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760943.902184:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512801920:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789760944.925044:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760945.949040:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760946.973033:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760947.997059:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760949.021026:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760949.021035:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789760949.021038:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760949.021047:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 923 00000100:00080000:1.0:1789760949.021055:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760949.021062:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789760949.021073:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789760949.021094:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512802048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760949.021114:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760949.021117:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760949.021118:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760949.021122:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 923 00000100:00080000:1.0:1789760949.021125:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760949.021129:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789760949.021136:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789760949.021145:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512802048:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789760949.021148:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512802176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760949.021160:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760949.021163:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789760949.021165:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760949.021168:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 923 00000100:00080000:1.0:1789760949.021170:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760949.021173:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789760949.021176:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789760949.021186:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512802304/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760949.021193:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760949.021196:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760949.021197:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760949.021200:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 923 00000100:00080000:1.0:1789760949.021203:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760949.021206:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789760949.021209:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760949.021219:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512802432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789760949.021255:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512802176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789760949.021301:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512802304:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789760949.021320:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512802432:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789760950.045102:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760951.069069:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760952.093020:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789760952.093192:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:2.0:1789760952.093236:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789760952.093252:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789760952.093292:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789760952.093301:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789760952.093312:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:3.0:1789760952.093328:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760949/real 1789760952] req@ffff8dc005c94700 x1876699512802176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760952.093360:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512802176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760952.093370:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760952.093376:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760952.093381:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512802176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789760952.093390:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760949/real 1789760952] req@ffff8dc005c94380 x1876699512802304/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760952.093404:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512802304/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760952.093414:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760952.093418:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760952.093422:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512802304:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789760952.093429:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760949/real 1789760952] req@ffff8dc005c97b80 x1876699512802432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760952.093443:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512802432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760952.093452:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760952.093456:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760952.093460:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512802432:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789760952.093467:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760949/real 1789760952] req@ffff8dc005c96300 x1876699512802048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760952.093480:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512802048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760952.093488:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760952.093492:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760952.093498:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512802048:192.168.203.156@tcp:250:kworker.0 00010000:00010000:2.0:1789760953.117043:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760954.140990:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760954.141001:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789760954.141006:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760954.141016:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 928 00010000:00010000:2.0:1789760954.141018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760954.141024:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760954.141031:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789760954.141061:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512802560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760954.141081:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760954.141084:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760954.141085:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760954.141088:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 928 00000100:00080000:1.0:1789760954.141093:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760954.141098:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789760954.141108:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760954.141114:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512802560:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789760954.141125:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512802688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760954.141138:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760954.141142:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789760954.141143:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760954.141147:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 928 00000100:00080000:1.0:1789760954.141152:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760954.141155:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789760954.141162:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1789760954.141168:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789760954.141179:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512802816/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760954.141190:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760954.141194:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760954.141196:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760954.141201:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 928 00000100:00080000:1.0:1789760954.141205:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760954.141210:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789760954.141217:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760954.141230:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760954/real 1789760954] req@ffff8dc005c94e00 x1876699512802560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789760954.141232:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512802944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760954.141275:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512802560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760954.141288:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760954.141292:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760954.141296:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512802560:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789760954.141321:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512802688:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760954.141345:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760954.141360:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760954/real 1789760954] req@ffff8dc005c97480 x1876699512802688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760954.141372:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512802688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760954.141381:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760954.141384:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760954.141387:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512802688:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789760954.141393:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512802816:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789760954.141414:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760954.141426:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760954/real 1789760954] req@ffff8dc005c96680 x1876699512802816/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760954.141437:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512802816/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760954.141446:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760954.141449:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760954.141452:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512802816:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789760954.141458:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512802944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760954.141482:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760954.141495:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760954/real 1789760954] req@ffff8dc005c95880 x1876699512802944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760954.141507:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512802944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760954.141515:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760954.141518:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760954.141522:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512802944:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789760955.165009:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760956.189039:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760957.213087:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760958.237060:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760959.261086:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1789760959.261086:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789760959.261101:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789760959.261106:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760959.261128:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 933 00000100:00080000:1.0:1789760959.261160:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760959.261169:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789760959.261228:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96d80 x1876699512803072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760959.261275:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760959.261282:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760959.261285:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760959.261293:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 933 00000100:00080000:1.0:1789760959.261301:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760959.261309:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789760959.261335:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760959.261342:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512803072:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789760959.261359:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512803200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760959.261373:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760959.261378:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789760959.261381:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760959.261386:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 933 00000100:00080000:1.0:1789760959.261401:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760959.261405:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789760959.261411:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789760959.261425:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512803328/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789760959.261452:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789760959.261456:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789760959.261458:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789760959.261464:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 933 00000800:00000100:3.0:1789760959.261469:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789760959.261471:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789760959.261479:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789760959.261485:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789760959.261496:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512803456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760959.261544:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760959/real 1789760959] req@ffff8dc005c96d80 x1876699512803072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760959.261573:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96d80 x1876699512803072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760959.261587:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760959.261594:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760959.261599:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512803072:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789760959.261643:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512803200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760959.261672:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760959.261687:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760959/real 1789760959] req@ffff8dc005c95880 x1876699512803200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760959.261703:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512803200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760959.261713:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760959.261718:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760959.261722:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512803200:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789760959.261730:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512803328:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789760959.261765:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760959.261780:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760959/real 1789760959] req@ffff8dc005c96680 x1876699512803328/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760959.261810:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512803328/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760959.261822:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760959.261826:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760959.261830:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512803328:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789760959.261849:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512803456:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760959.261873:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760959.261888:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760959/real 1789760959] req@ffff8dc005c97480 x1876699512803456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760959.261929:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512803456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760959.261939:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760959.261943:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760959.261947:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512803456:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789760960.285042:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760961.309016:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760962.333044:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760963.357193:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760964.381115:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760964.381123:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1789760964.381126:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760964.381127:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760964.381138:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 938 00000100:00080000:0.0:1789760964.381146:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760964.381152:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789760964.381186:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514a680 x1876699512803584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760964.381217:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760964.381220:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760964.381222:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760964.381225:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 938 00000100:00080000:0.0:1789760964.381227:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760964.381230:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760964.381237:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760964.381264:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149880 x1876699512803712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760964.381271:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760964.381274:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760964.381276:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760964.381278:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 938 00000100:00080000:0.0:1789760964.381281:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789760964.381285:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512803584:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789760964.381286:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760964.381293:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760964.381300:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b800 x1876699512803840/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760964.381308:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760964.381311:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760964.381312:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760964.381316:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 938 00000100:00080000:0.0:1789760964.381321:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760964.381325:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760964.381330:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760964.381359:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149f80 x1876699512803968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789760964.381366:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760964.381425:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760964/real 1789760964] req@ffff8dc00514a680 x1876699512803584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760964.381442:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514a680 x1876699512803584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760964.381467:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760964.381470:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760964.381473:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512803584:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789760964.381478:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512803712:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760964.381495:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760964.381503:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760964/real 1789760964] req@ffff8dc005149880 x1876699512803712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760964.381517:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149880 x1876699512803712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760964.381522:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760964.381525:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760964.381527:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512803712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789760964.381544:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512803840:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789760964.381557:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760964.381564:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760964/real 1789760964] req@ffff8dc00514b800 x1876699512803840/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760964.381571:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b800 x1876699512803840/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760964.381577:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760964.381579:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760964.381581:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512803840:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789760964.381585:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512803968:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760964.381595:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760964.381601:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760964/real 1789760964] req@ffff8dc005149f80 x1876699512803968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760964.381609:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149f80 x1876699512803968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760964.381614:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760964.381615:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760964.381617:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512803968:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789760965.405033:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760966.429116:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760967.453046:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760968.477047:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760969.501074:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760969.501088:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760969.501094:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1789760969.501096:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760969.501113:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 944 00000100:00080000:0.0:1789760969.501127:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760969.501134:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789760969.501189:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149180 x1876699512804096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760969.501214:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760969.501217:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760969.501219:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760969.501222:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 944 00000100:00080000:0.0:1789760969.501226:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760969.501230:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760969.501240:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760969.501251:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149c00 x1876699512804224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760969.501257:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512804096:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789760969.501272:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760969.501276:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760969.501277:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760969.501280:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 944 00000100:00080000:0.0:1789760969.501284:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760969.501287:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760969.501291:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760969.501319:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b480 x1876699512804352/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760969.501327:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760969.501331:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760969.501333:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760969.501336:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 944 00000100:00080000:0.0:1789760969.501340:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789760969.501348:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1789760969.501353:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760969.501381:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760969.501402:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514bb80 x1876699512804480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760969.501412:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760969/real 1789760969] req@ffff8dc005149180 x1876699512804096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760969.501434:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149180 x1876699512804096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760969.501444:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760969.501448:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760969.501452:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512804096:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789760969.501471:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512804224:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760969.501487:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760969.501496:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760969/real 1789760969] req@ffff8dc005149c00 x1876699512804224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760969.501504:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149c00 x1876699512804224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760969.501509:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760969.501512:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760969.501514:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512804224:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789760969.501527:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512804352:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789760969.501538:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760969.501544:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760969/real 1789760969] req@ffff8dc00514b480 x1876699512804352/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760969.501552:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b480 x1876699512804352/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760969.501557:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760969.501559:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760969.501561:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512804352:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789760969.501565:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512804480:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760969.501575:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760969.501581:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760969/real 1789760969] req@ffff8dc00514bb80 x1876699512804480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760969.501588:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514bb80 x1876699512804480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760969.501594:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760969.501596:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760969.501598:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512804480:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789760970.525096:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760971.549061:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760972.573100:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760973.597091:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760974.621000:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760974.621009:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760974.621013:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760974.621023:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 949 00000100:00080000:0.0:1789760974.621032:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760974.621050:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789760974.621084:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148380 x1876699512804608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760974.621107:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760974.621110:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760974.621111:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760974.621114:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 949 00000100:00080000:0.0:1789760974.621117:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760974.621120:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760974.621128:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760974.621139:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512804608:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789760974.621140:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148000 x1876699512804736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760974.621154:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760974.621159:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760974.621160:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760974.621164:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 949 00000100:00080000:0.0:1789760974.621166:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760974.621168:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760974.621172:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760974.621181:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514ad80 x1876699512804864/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760974.621189:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760974.621199:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1789760974.621200:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1789760974.621202:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760974.621208:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 949 00000100:00080000:0.0:1789760974.621212:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760974.621216:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760974.621220:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760974.621230:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148700 x1876699512804992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760974.621249:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760974/real 1789760974] req@ffff8dc005148380 x1876699512804608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760974.621265:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148380 x1876699512804608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789760999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760974.621274:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760974.621278:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760974.621282:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512804608:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789760974.621317:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512804736:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789760974.621320:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:3.0:1789760974.621335:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760974.621354:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760974/real 1789760974] req@ffff8dc005148000 x1876699512804736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760974.621364:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148000 x1876699512804736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760974.621369:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760974.621383:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760974.621386:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512804736:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789760974.621390:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512804864:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789760974.621409:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760974.621420:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760974/real 1789760974] req@ffff8dc00514ad80 x1876699512804864/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760974.621434:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514ad80 x1876699512804864/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789760999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760974.621443:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760974.621446:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760974.621450:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512804864:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789760974.621457:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512804992:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760974.621489:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760974.621502:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760974/real 1789760974] req@ffff8dc005148700 x1876699512804992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760974.621515:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148700 x1876699512804992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789760999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760974.621523:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760974.621527:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760974.621530:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512804992:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789760975.645085:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760976.669150:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760977.693051:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760978.717040:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760979.741033:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760979.741042:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760979.741046:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760979.741055:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 954 00000100:00080000:0.0:1789760979.741064:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760979.741070:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789760979.741102:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148a80 x1876699512805120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760979.741124:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760979.741127:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760979.741132:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760979.741136:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 954 00000100:00080000:0.0:1789760979.741155:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760979.741159:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760979.741168:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760979.741177:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514aa00 x1876699512805248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760979.741183:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512805120:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789760979.741187:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760979.741193:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760979.741195:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760979.741198:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 954 00000100:00080000:0.0:1789760979.741200:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760979.741202:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760979.741206:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760979.741216:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148700 x1876699512805376/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760979.741223:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760979.741225:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760979.741227:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760979.741229:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 954 00000100:00080000:0.0:1789760979.741234:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760979.741236:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760979.741239:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760979.741246:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514ad80 x1876699512805504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789760979.741256:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760979.741297:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760979/real 1789760979] req@ffff8dc005148a80 x1876699512805120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760979.741312:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148a80 x1876699512805120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760979.741321:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760979.741324:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760979.741328:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512805120:192.168.203.156@tcp:250:kworker.0 00010000:00010000:1.0:1789760979.741329:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789760979.741349:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512805248:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760979.741367:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760979.741379:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760979/real 1789760979] req@ffff8dc00514aa00 x1876699512805248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760979.741393:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514aa00 x1876699512805248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760979.741401:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760979.741405:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760979.741409:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512805248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789760979.741415:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512805376:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789760979.741434:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760979.741442:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760979/real 1789760979] req@ffff8dc005148700 x1876699512805376/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760979.741454:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148700 x1876699512805376/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760979.741460:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760979.741462:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760979.741464:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512805376:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789760979.741467:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512805504:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760979.741478:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760979.741485:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760979/real 1789760979] req@ffff8dc00514ad80 x1876699512805504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760979.741492:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514ad80 x1876699512805504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760979.741497:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760979.741498:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760979.741500:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512805504:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789760980.765088:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789760981.789059:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760982.813081:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760983.837130:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760984.861009:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760984.861017:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760984.861021:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760984.861033:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 959 00000100:00080000:0.0:1789760984.861041:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760984.861048:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789760984.861081:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148000 x1876699512805632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760984.861104:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760984.861107:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760984.861110:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760984.861113:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 959 00000100:00080000:0.0:1789760984.861116:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760984.861120:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760984.861128:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760984.861137:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148380 x1876699512805760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760984.861145:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760984.861147:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760984.861149:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760984.861152:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 959 00000100:00080000:0.0:1789760984.861154:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760984.861156:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760984.861159:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760984.861167:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514bb80 x1876699512805888/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760984.861173:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760984.861175:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760984.861176:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760984.861179:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 959 00000100:00080000:0.0:1789760984.861181:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760984.861183:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760984.861186:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760984.861195:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b480 x1876699512806016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789760984.861248:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512805632:192.168.203.156@tcp:250:kworker.0 00010000:00010000:0.0:1789760984.861279:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:3.0:1789760984.861306:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760984.861353:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760984/real 1789760984] req@ffff8dc005148000 x1876699512805632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760984.861426:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148000 x1876699512805632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760984.861440:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760984.861443:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760984.861446:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512805632:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789760984.861451:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512805760:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760984.861467:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760984.861476:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760984/real 1789760984] req@ffff8dc005148380 x1876699512805760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760984.861587:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148380 x1876699512805760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760984.861593:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760984.861595:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760984.861599:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512805760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789760984.861652:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512805888:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789760984.861673:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760984.861687:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760984/real 1789760984] req@ffff8dc00514bb80 x1876699512805888/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760984.861707:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514bb80 x1876699512805888/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760984.861713:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760984.861715:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760984.861717:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512805888:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789760984.861721:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512806016:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760984.861732:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760984.861738:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760984/real 1789760984] req@ffff8dc00514b480 x1876699512806016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760984.861745:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b480 x1876699512806016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760984.861750:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760984.861752:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760984.861754:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512806016:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789760985.885036:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760986.909018:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760987.933063:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760988.957077:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760989.981014:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760989.981022:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760989.981026:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760989.981036:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 964 00000100:00080000:0.0:1789760989.981047:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760989.981055:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789760989.981086:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149c00 x1876699512806144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760989.981109:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760989.981111:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760989.981113:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760989.981116:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 964 00000100:00080000:0.0:1789760989.981123:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760989.981127:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760989.981135:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760989.981144:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149180 x1876699512806272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760989.981152:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760989.981155:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760989.981157:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760989.981161:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 964 00000100:00080000:0.0:1789760989.981165:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760989.981169:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789760989.981169:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512806144:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1789760989.981172:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760989.981184:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149f80 x1876699512806400/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760989.981197:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760989.981199:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760989.981200:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760989.981204:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 964 00000100:00080000:0.0:1789760989.981206:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760989.981208:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760989.981211:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760989.981218:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b800 x1876699512806528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1789760989.981259:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:3.0:1789760989.981272:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760989.981335:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760989/real 1789760989] req@ffff8dc005149c00 x1876699512806144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760989.981357:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149c00 x1876699512806144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760989.981371:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760989.981378:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760989.981383:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512806144:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789760989.981436:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512806272:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760989.981465:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760989.981481:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760989/real 1789760989] req@ffff8dc005149180 x1876699512806272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760989.981497:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149180 x1876699512806272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760989.981507:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760989.981512:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760989.981516:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512806272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789760989.981524:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512806400:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789760989.981549:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760989.981580:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760989/real 1789760989] req@ffff8dc005149f80 x1876699512806400/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760989.981599:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149f80 x1876699512806400/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760989.981608:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760989.981629:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760989.981632:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512806400:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789760989.981639:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512806528:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760989.981661:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760989.981673:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760989/real 1789760989] req@ffff8dc00514b800 x1876699512806528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760989.981685:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b800 x1876699512806528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760989.981702:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760989.981706:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760989.981710:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512806528:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789760991.005016:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760992.029115:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760993.053026:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760994.077049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789760995.100980:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760995.100997:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789760995.101001:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760995.101012:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 969 00000100:00080000:0.0:1789760995.101023:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760995.101030:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789760995.101076:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149880 x1876699512806656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760995.101101:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760995.101104:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760995.101107:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760995.101110:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 969 00000100:00080000:0.0:1789760995.101113:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760995.101117:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760995.101132:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789760995.101137:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512806656:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789760995.101144:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514a680 x1876699512806784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760995.101154:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760995.101156:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789760995.101157:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760995.101161:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 969 00000100:00080000:0.0:1789760995.101163:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760995.101165:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760995.101169:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789760995.101183:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514a300 x1876699512806912/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789760995.101191:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789760995.101194:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789760995.101195:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789760995.101198:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 969 00000100:00080000:0.0:1789760995.101200:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789760995.101203:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789760995.101207:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789760995.101215:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148e00 x1876699512807040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789760995.101220:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760995.101288:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760995/real 1789760995] req@ffff8dc005149880 x1876699512806656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760995.101307:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149880 x1876699512806656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760995.101317:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00010000:00010000:1.0:1789760995.101320:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789760995.101322:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760995.101329:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512806656:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789760995.101352:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512806784:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760995.101367:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760995.101375:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760995/real 1789760995] req@ffff8dc00514a680 x1876699512806784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760995.101384:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514a680 x1876699512806784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760995.101389:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760995.101392:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760995.101394:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512806784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789760995.101399:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512806912:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789760995.101411:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760995.101417:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760995/real 1789760995] req@ffff8dc00514a300 x1876699512806912/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760995.101425:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514a300 x1876699512806912/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760995.101431:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760995.101432:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760995.101434:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512806912:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789760995.101438:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512807040:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789760995.101449:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789760995.101455:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789760995/real 1789760995] req@ffff8dc005148e00 x1876699512807040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789760995.101462:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148e00 x1876699512807040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789760995.101467:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789760995.101469:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789760995.101470:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512807040:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789760996.125021:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789760997.149000:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789760998.173103:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789760999.197051:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761000.221055:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761000.221068:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761000.221074:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761000.221088:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 974 00000100:00080000:1.0:1789761000.221102:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761000.221111:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761000.221120:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761000.221158:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512807168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761000.221184:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761000.221187:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761000.221189:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761000.221192:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 974 00000100:00080000:1.0:1789761000.221196:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761000.221201:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761000.221214:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761000.221233:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512807296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761000.221233:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512807168:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761000.221254:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761000.221257:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761000.221260:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761000.221265:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 974 00000100:00080000:1.0:1789761000.221270:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761000.221274:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761000.221280:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761000.221298:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512807424/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761000.221311:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761000.221315:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761000.221317:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761000.221323:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 974 00000800:00000100:3.0:1789761000.221327:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761000.221330:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761000.221337:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761000.221343:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761000.221356:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512807552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761000.221396:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761000/real 1789761000] req@ffff8dc005c94e00 x1876699512807168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761000.221430:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512807168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761000.221446:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761000.221452:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761000.221457:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512807168:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761000.221503:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512807296:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761000.221532:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761000.221548:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761000/real 1789761000] req@ffff8dc005c96300 x1876699512807296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761000.221564:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512807296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761000.221573:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761000.221576:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761000.221580:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512807296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761000.221588:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512807424:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761000.221614:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761000.221627:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761000/real 1789761000] req@ffff8dc005c97b80 x1876699512807424/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761000.221642:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512807424/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761000.221660:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761000.221664:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761000.221668:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512807424:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761000.221676:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512807552:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761000.221700:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761000.221713:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761000/real 1789761000] req@ffff8dc005c94380 x1876699512807552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761000.221745:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512807552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761000.221755:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761000.221759:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761000.221762:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512807552:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761001.245047:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761002.269030:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761003.293106:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761004.317179:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761005.341011:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761005.341019:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761005.341022:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761005.341031:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 979 00010000:00010000:0.0:1789761005.341034:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761005.341041:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761005.341048:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761005.341079:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512807680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761005.341105:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761005.341108:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761005.341109:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761005.341112:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 979 00000100:00080000:1.0:1789761005.341115:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761005.341118:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761005.341126:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761005.341135:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512807808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761005.341143:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761005.341146:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1789761005.341146:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512807680:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761005.341151:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761005.341156:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 979 00000100:00080000:1.0:1789761005.341158:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761005.341161:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761005.341163:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761005.341173:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512807936/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761005.341184:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761005.341187:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761005.341189:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761005.341193:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 979 00000100:00080000:1.0:1789761005.341197:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761005.341201:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:3.0:1789761005.341206:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1789761005.341208:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761005.341225:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512808064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761005.341243:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761005/real 1789761005] req@ffff8dc005c94700 x1876699512807680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761005.341258:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512807680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761005.341265:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761005.341268:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761005.341271:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512807680:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761005.341288:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512807808:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761005.341302:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761005.341309:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761005/real 1789761005] req@ffff8dc005c96a00 x1876699512807808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761005.341317:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512807808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761005.341322:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761005.341325:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761005.341327:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512807808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761005.341331:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512807936:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761005.341342:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761005.341347:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761005/real 1789761005] req@ffff8dc005c95500 x1876699512807936/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761005.341355:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512807936/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761005.341360:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761005.341362:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761005.341363:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512807936:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761005.341367:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512808064:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761005.341377:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761005.341383:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761005/real 1789761005] req@ffff8dc005c95f80 x1876699512808064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761005.341390:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512808064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761005.341395:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761005.341397:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761005.341399:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512808064:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761006.365033:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761007.389084:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761008.413050:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761009.437073:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761010.461040:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761010.461050:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761010.461056:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761010.461069:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 985 00000100:00080000:0.0:1789761010.461080:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761010.461088:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761010.461133:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149500 x1876699512808192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761010.461159:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761010.461161:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761010.461163:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761010.461166:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 985 00000100:00080000:0.0:1789761010.461168:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761010.461171:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761010.461180:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761010.461191:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b100 x1876699512808320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761010.461200:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761010.461203:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761010.461205:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761010.461214:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 985 00000100:00080000:0.0:1789761010.461217:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789761010.461218:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512808192:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789761010.461221:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761010.461233:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761010.461249:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148e00 x1876699512808448/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761010.461257:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761010.461260:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761010.461261:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761010.461265:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 985 00000100:00080000:0.0:1789761010.461268:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761010.461272:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761010.461275:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761010.461282:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514a300 x1876699512808576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761010.461295:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761010.461351:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761010/real 1789761010] req@ffff8dc005149500 x1876699512808192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1789761010.461363:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761010.461373:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149500 x1876699512808192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761010.461382:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761010.461386:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761010.461389:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512808192:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761010.461410:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512808320:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761010.461444:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761010.461453:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761010/real 1789761010] req@ffff8dc00514b100 x1876699512808320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761010.461467:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b100 x1876699512808320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761010.461490:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761010.461495:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761010.461498:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512808320:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761010.461506:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512808448:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761010.461529:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761010.461542:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761010/real 1789761010] req@ffff8dc005148e00 x1876699512808448/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761010.461555:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148e00 x1876699512808448/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761010.461562:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761010.461565:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761010.461567:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512808448:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761010.461572:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512808576:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761010.461588:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761010.461596:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761010/real 1789761010] req@ffff8dc00514a300 x1876699512808576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761010.461603:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514a300 x1876699512808576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761010.461608:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761010.461611:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761010.461613:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512808576:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761011.485077:0:11139:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 3/3 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1789761011.485104:0:5597:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 3 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1789761011.485114:0:5597:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8dc013c56000 lock: ffff8dc0029d6c00/0xe9383def1c124b52 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xe6aa143190c2410e expref: -99 pid: 11789 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1789761011.485127:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:0.0:1789761011.485157:0:5597:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x9:0x0]: losing layout lock 00010000:00010000:0.0:1789761011.485178:0:5597:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8dc013c56000 lock: ffff8dc00376ec00/0xe9383def1c124b4b lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe6aa143190c240f2 expref: -99 pid: 11789 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789761011.485196:0:5597:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8dc013c56000 lock: ffff8dc000031400/0xe9383def1c124b36 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xe6aa143190c24090 expref: -99 pid: 11782 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789761011.485251:0:5597:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8dc013c56000 lock: ffff8dc0029d6c00/0xe9383def1c124b52 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe6aa143190c2410e expref: -99 pid: 11789 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789761011.485263:0:5597:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8dc013c56000 lock: ffff8dc00376ec00/0xe9383def1c124b4b lrc: 1/0,0 mode: --/PR res: [0x200000401:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe6aa143190c240f2 expref: -99 pid: 11789 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789761011.485268:0:5597:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8dc013c56000 lock: ffff8dc000031400/0xe9383def1c124b36 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe6aa143190c24090 expref: -99 pid: 11782 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789761011.485273:0:5597:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1789761011.485300:0:5597:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8dc013c56000 lock: ffff8dc0029d6c00/0xe9383def1c124b52 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xe6aa143190c2410e expref: -99 pid: 11789 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789761011.485324:0:5597:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8dc013c56000 lock: ffff8dc00376ec00/0xe9383def1c124b4b lrc: 0/0,0 mode: --/PR res: [0x200000401:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe6aa143190c240f2 expref: -99 pid: 11789 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1789761011.485334:0:5597:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8dc013c56000 lock: ffff8dc000031400/0xe9383def1c124b36 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xe6aa143190c24090 expref: -99 pid: 11782 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1789761011.485346:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761011.485349:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761011.485351:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761011.485357:0:2359:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8dc00514a680 x1876699512808704/t0(0) o103->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:17/18 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1789761012.509068:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761013.533062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761014.557071:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761015.581035:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761015.581050:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761015.581055:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761015.581068:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 990 00000100:00080000:1.0:1789761015.581083:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1789761015.581086:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761015.581094:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761015.581156:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512808832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761015.581188:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761015.581191:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761015.581193:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761015.581196:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 990 00000100:00080000:1.0:1789761015.581199:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761015.581202:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761015.581214:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761015.581232:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512808960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761015.581237:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512808832:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761015.581254:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761015.581260:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761015.581262:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761015.581266:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 990 00000100:00080000:1.0:1789761015.581269:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761015.581273:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761015.581278:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761015.581292:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512809088/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761015.581300:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761015.581303:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761015.581304:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761015.581307:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 990 00000100:00080000:1.0:1789761015.581309:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761015.581312:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761015.581315:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761015.581326:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512809216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761015.581359:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512808960:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761015.581413:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512809088:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761015.581444:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512809216:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761016.605071:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761017.629025:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761018.653115:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789761018.653195:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1789761018.653237:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789761018.653252:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761018.653303:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761018.653317:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:3.0:1789761018.653340:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761015/real 1789761018] req@ffff8dc005c97800 x1876699512808960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1789761018.653346:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:3.0:1789761018.653385:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512808960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761018.653399:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761018.653404:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761018.653409:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512808960:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761018.653427:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761015/real 1789761018] req@ffff8dc005c94a80 x1876699512809088/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761018.653440:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512809088/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761018.653450:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761018.653454:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761018.653458:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512809088:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761018.653467:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761015/real 1789761018] req@ffff8dc005c95180 x1876699512809216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761018.653479:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512809216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761018.653487:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761018.653490:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761018.653493:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512809216:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761018.653501:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761015/real 1789761018] req@ffff8dc005c95c00 x1876699512808832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761018.653513:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512808832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761018.653521:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761018.653524:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761018.653528:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512808832:192.168.203.156@tcp:250:kworker.0 00010000:00010000:1.0:1789761019.677081:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761020.701036:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761020.701045:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761020.701048:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761020.701057:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 995 00000100:00080000:2.0:1789761020.701065:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761020.701075:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761020.701112:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512809344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1789761020.701119:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761020.701142:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761020.701147:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761020.701149:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761020.701153:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 995 00000100:00080000:2.0:1789761020.701159:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761020.701165:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761020.701175:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761020.701180:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512809344:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761020.701194:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512809472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761020.701207:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761020.701210:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761020.701212:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761020.701217:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 995 00000100:00080000:2.0:1789761020.701223:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761020.701227:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761020.701232:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761020.701250:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512809600/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761020.701262:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789761020.701267:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761020.701272:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761020.701275:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761020.701283:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 995 00000100:00080000:2.0:1789761020.701289:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761020.701296:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761020.701302:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761020.701311:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761020/real 1789761020] req@ffff8dc00368dc00 x1876699512809344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761020.701320:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512809728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761020.701372:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512809344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761020.701385:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761020.701391:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761020.701395:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512809344:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761020.701422:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512809472:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761020.701453:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761020.701468:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761020/real 1789761020] req@ffff8dc00368c700 x1876699512809472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761020.701482:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512809472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761020.701491:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761020.701495:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761020.701499:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512809472:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761020.701506:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512809600:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761020.701539:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761020.701555:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761020/real 1789761020] req@ffff8dc00368d180 x1876699512809600/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761020.701568:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512809600/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761020.701584:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761020.701588:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761020.701592:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512809600:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761020.701598:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512809728:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761020.701618:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761020.701631:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761020/real 1789761020] req@ffff8dc00368e680 x1876699512809728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761020.701643:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512809728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761020.701651:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761020.701654:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761020.701658:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512809728:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761021.725074:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761022.749086:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761023.773055:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761024.797047:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761025.821051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761025.821064:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761025.821071:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761025.821076:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761025.821084:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1000 00000100:00080000:2.0:1789761025.821092:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761025.821097:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761025.821130:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512809856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761025.821149:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761025.821152:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761025.821153:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761025.821156:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1000 00000100:00080000:2.0:1789761025.821159:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761025.821162:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761025.821169:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761025.821180:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512809984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761025.821185:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512809856:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789761025.821197:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761025.821202:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761025.821204:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761025.821209:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1000 00000100:00080000:2.0:1789761025.821214:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761025.821218:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761025.821225:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761025.821239:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512810112/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761025.821251:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789761025.821252:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761025.821258:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761025.821261:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761025.821267:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1000 00000100:00080000:2.0:1789761025.821274:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761025.821278:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761025.821283:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761025.821294:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761025/real 1789761025] req@ffff8dc00368df80 x1876699512809856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761025.821314:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512810240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761025.821351:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512809856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761025.821372:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761025.821376:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761025.821381:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512809856:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761025.821404:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512809984:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761025.821425:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761025.821437:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761025/real 1789761025] req@ffff8dc00368e300 x1876699512809984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761025.821450:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512809984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761025.821458:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761025.821462:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761025.821465:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512809984:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761025.821471:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512810112:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761025.821490:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761025.821501:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761025/real 1789761025] req@ffff8dc00368ed80 x1876699512810112/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761025.821513:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512810112/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761025.821522:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761025.821525:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761025.821528:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512810112:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761025.821535:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512810240:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761025.821551:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761025.821561:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761025/real 1789761025] req@ffff8dc00368f800 x1876699512810240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761025.821574:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512810240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761025.821582:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761025.821586:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761025.821589:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512810240:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761026.845074:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761027.869045:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761028.893068:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761029.917033:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761030.941000:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761030.941010:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761030.941015:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761030.941027:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1005 00000100:00080000:3.0:1789761030.941037:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761030.941045:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789761030.941063:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761030.941076:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512810368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761030.941102:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761030.941105:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761030.941106:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761030.941109:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1005 00000100:00080000:3.0:1789761030.941112:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761030.941114:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761030.941121:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761030.941130:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512810496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761030.941137:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761030.941140:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761030.941141:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761030.941144:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1005 00000100:00080000:3.0:1789761030.941146:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761030.941152:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512810368:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761030.941152:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761030.941159:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761030.941169:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512810624/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761030.941176:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761030.941178:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761030.941180:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761030.941182:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1005 00000100:00080000:3.0:1789761030.941185:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761030.941188:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761030.941191:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761030.941199:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512810752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761030.941225:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761030.941280:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761030/real 1789761030] req@ffff8dc0107ace00 x1876699512810368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761030.941297:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512810368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761030.941305:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761030.941309:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761030.941312:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512810368:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761030.941318:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512810496:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761030.941335:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761030.941344:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761030/real 1789761030] req@ffff8dc0107af100 x1876699512810496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761030.941356:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512810496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761030.941364:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761030.941367:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761030.941371:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512810496:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761030.941396:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512810624:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761030.941416:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761030.941424:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761030/real 1789761030] req@ffff8dc0107ad880 x1876699512810624/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761030.941432:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512810624/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761030.941438:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761030.941440:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761030.941442:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512810624:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761030.941445:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512810752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761030.941456:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761030.941462:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761030/real 1789761030] req@ffff8dc0107aea00 x1876699512810752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761030.941470:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512810752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761030.941474:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761030.941476:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761030.941478:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512810752:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761031.965037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761032.989028:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761034.013030:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761035.037127:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761036.060972:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761036.060987:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761036.060991:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761036.060999:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1010 00000100:00080000:3.0:1789761036.061007:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761036.061012:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761036.061044:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512810880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1789761036.061049:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761036.061063:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761036.061066:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761036.061068:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761036.061071:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1010 00000100:00080000:3.0:1789761036.061074:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761036.061078:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761036.061080:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512810880:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789761036.061086:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761036.061097:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512811008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761036.061105:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761036.061108:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761036.061110:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761036.061112:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1010 00000100:00080000:3.0:1789761036.061116:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761036.061118:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761036.061121:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761036.061128:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512811136/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761036.061135:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789761036.061138:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761036.061140:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761036.061142:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761036.061147:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1010 00000100:00080000:3.0:1789761036.061150:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761036.061153:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761036.061156:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761036.061164:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512811264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761036.061187:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761036/real 1789761036] req@ffff8dc0107ae680 x1876699512810880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761036.061202:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512810880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761036.061210:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761036.061213:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761036.061216:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512810880:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761036.061244:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512811008:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761036.061259:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761036.061267:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761036/real 1789761036] req@ffff8dc0107af800 x1876699512811008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761036.061280:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512811008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761036.061286:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761036.061288:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761036.061291:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512811008:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761036.061295:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512811136:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761036.061306:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761036.061312:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761036/real 1789761036] req@ffff8dc0107adf80 x1876699512811136/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761036.061319:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512811136/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761036.061324:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761036.061325:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761036.061327:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512811136:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761036.061331:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512811264:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761036.061341:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761036.061347:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761036/real 1789761036] req@ffff8dc0107af480 x1876699512811264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761036.061354:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512811264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761036.061363:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761036.061365:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761036.061367:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512811264:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761037.085101:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761038.109097:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761039.133099:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761040.157010:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761041.181005:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1789761041.181014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761041.181014:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761041.181017:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761041.181027:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1015 00000100:00080000:3.0:1789761041.181035:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761041.181041:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761041.181075:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512811392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761041.181091:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761041.181095:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761041.181097:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761041.181101:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1015 00000100:00080000:3.0:1789761041.181104:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761041.181107:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761041.181115:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761041.181127:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512811392:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761041.181127:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512811520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761041.181143:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761041.181146:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761041.181147:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761041.181150:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1015 00000100:00080000:3.0:1789761041.181153:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761041.181156:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761041.181160:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761041.181171:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512811648/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761041.181179:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761041.181195:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761041.181197:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761041.181203:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1015 00000800:00000100:1.0:1789761041.181209:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761041.181212:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761041.181220:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761041.181226:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761041.181241:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512811776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761041.181257:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761041/real 1789761041] req@ffff8dc0107ad180 x1876699512811392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761041.181279:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512811392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761041.181291:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761041.181296:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761041.181300:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512811392:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761041.181320:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512811520:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761041.181339:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761041.181348:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761041/real 1789761041] req@ffff8dc0107ac380 x1876699512811520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761041.181356:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512811520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761041.181362:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761041.181365:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761041.181367:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512811520:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761041.181371:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512811648:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761041.181382:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761041.181387:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761041/real 1789761041] req@ffff8dc0107aed80 x1876699512811648/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761041.181395:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512811648/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761041.181400:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761041.181401:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761041.181403:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512811648:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761041.181406:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512811776:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761041.181416:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761041.181422:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761041/real 1789761041] req@ffff8dc0107ad500 x1876699512811776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761041.181429:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512811776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761041.181434:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761041.181436:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761041.181438:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512811776:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761042.205046:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761043.229026:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761044.253085:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761045.277120:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761046.301037:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761046.301054:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761046.301061:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761046.301076:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1020 00000100:00080000:1.0:1789761046.301091:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761046.301103:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761046.301109:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761046.301175:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512811904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761046.301212:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761046.301217:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761046.301219:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761046.301224:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1020 00000100:00080000:1.0:1789761046.301230:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761046.301236:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761046.301249:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761046.301255:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512811904:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761046.301272:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512812032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761046.301285:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761046.301289:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761046.301292:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761046.301296:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1020 00000100:00080000:1.0:1789761046.301302:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761046.301306:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761046.301314:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761046.301329:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512812160/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761046.301341:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789761046.301346:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761046.301348:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761046.301353:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761046.301358:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1020 00000100:00080000:1.0:1789761046.301363:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761046.301369:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761046.301375:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761046.301391:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512812288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761046.301401:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761046/real 1789761046] req@ffff8dc005c97100 x1876699512811904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761046.301420:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512811904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761046.301429:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761046.301433:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761046.301437:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512811904:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761046.301462:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512812032:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761046.301479:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761046.301488:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761046/real 1789761046] req@ffff8dc005c95c00 x1876699512812032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761046.301497:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512812032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761046.301503:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761046.301505:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761046.301508:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512812032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761046.301512:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512812160:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761046.301525:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761046.301531:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761046/real 1789761046] req@ffff8dc005c95180 x1876699512812160/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761046.301539:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512812160/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761046.301544:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761046.301546:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761046.301549:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512812160:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761046.301552:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512812288:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761046.301562:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761046.301568:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761046/real 1789761046] req@ffff8dc005c94a80 x1876699512812288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761046.301576:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512812288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761046.301581:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761046.301583:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761046.301584:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512812288:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761047.325014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761048.349022:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761049.373043:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761050.397034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761051.421031:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761051.421039:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761051.421042:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761051.421050:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1026 00000100:00080000:2.0:1789761051.421059:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761051.421065:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789761051.421087:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789761051.421099:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512812416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761051.421135:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761051.421138:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761051.421139:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761051.421143:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1026 00000100:00080000:2.0:1789761051.421146:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761051.421149:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761051.421156:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761051.421168:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512812544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761051.421176:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761051.421178:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761051.421180:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761051.421183:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1026 00000100:00080000:2.0:1789761051.421185:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761051.421188:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761051.421192:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761051.421199:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512812416:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761051.421203:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512812672/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761051.421226:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761051.421228:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761051.421229:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761051.421233:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1026 00000100:00080000:2.0:1789761051.421235:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761051.421238:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761051.421241:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761051.421251:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512812800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761051.421308:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761051.421363:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761051/real 1789761051] req@ffff8dc00368ea00 x1876699512812416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761051.421384:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512812416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761051.421397:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761051.421402:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761051.421407:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512812416:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761051.421432:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512812544:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761051.421457:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761051.421473:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761051/real 1789761051] req@ffff8dc00368ce00 x1876699512812544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761051.421503:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512812544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761051.421513:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761051.421517:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761051.421520:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512812544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761051.421565:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512812672:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761051.421601:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761051.421617:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761051/real 1789761051] req@ffff8dc00368ca80 x1876699512812672/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761051.421647:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512812672/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761051.421670:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761051.421674:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761051.421678:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512812672:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761051.421684:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512812800:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761051.421705:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761051.421716:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761051/real 1789761051] req@ffff8dc00368c000 x1876699512812800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761051.421729:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512812800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761051.421764:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761051.421768:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761051.421772:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512812800:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761052.445066:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761053.469017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761054.493033:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761055.517018:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761056.541008:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761056.541019:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761056.541024:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761056.541038:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1031 00010000:00010000:0.0:1789761056.541046:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761056.541051:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761056.541078:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761056.541126:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512812928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761056.541153:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761056.541157:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761056.541159:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761056.541163:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1031 00000100:00080000:3.0:1789761056.541166:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761056.541169:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761056.541180:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761056.541197:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512813056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761056.541211:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761056.541218:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789761056.541221:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512812928:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761056.541221:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761056.541230:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1031 00000100:00080000:3.0:1789761056.541237:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761056.541243:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761056.541251:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761056.541268:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512813184/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761056.541305:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761056.541351:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761056/real 1789761056] req@ffff8dc0107aca80 x1876699512812928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761056.541371:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761056.541376:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761056.541382:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761056.541384:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512812928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761056.541393:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1031 00000100:00080000:3.0:1789761056.541402:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761056.541417:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761056.541425:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761056.541432:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512812928:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761056.541461:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512813056:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761056.541489:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761056.541506:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761056/real 1789761056] req@ffff8dc0107ac000 x1876699512813056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761056.541523:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512813056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761056.541535:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761056.541538:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1789761056.541546:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000008:00100000:3.0:1789761056.541550:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761056.541556:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512813056:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761056.541567:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512813184:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761056.541595:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761056.541608:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512813312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761056.541622:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761056/real 1789761056] req@ffff8dc0107ac700 x1876699512813184/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761056.541643:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512813184/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761056.541653:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761056.541658:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761056.541661:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512813184:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761056.541680:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512813312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761056.541705:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761056.541717:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761056/real 1789761056] req@ffff8dc0107ae300 x1876699512813312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761056.541730:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512813312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761056.541739:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761056.541743:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761056.541746:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512813312:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761057.565011:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761058.589074:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761059.613040:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761060.637099:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761061.661021:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761061.661046:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761061.661052:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761061.661066:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1036 00000100:00080000:3.0:1789761061.661079:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761061.661093:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761061.661100:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761061.661147:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699512813440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761061.661178:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761061.661184:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761061.661186:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761061.661192:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1036 00000100:00080000:3.0:1789761061.661198:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761061.661203:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761061.661214:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512813440:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789761061.661215:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761061.661236:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699512813568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761061.661250:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761061.661254:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761061.661256:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761061.661260:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1036 00000100:00080000:3.0:1789761061.661266:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761061.661271:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761061.661277:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761061.661291:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512813696/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761061.661299:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761061.661308:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761061.661312:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761061.661314:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761061.661319:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1036 00000100:00080000:3.0:1789761061.661324:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761061.661328:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761061.661334:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761061.661347:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512813824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761061.661358:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761061/real 1789761061] req@ffff8dc0107adc00 x1876699512813440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761061.661383:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699512813440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761061.661393:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761061.661396:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761061.661399:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512813440:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761061.661423:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512813568:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761061.661442:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761061.661451:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761061/real 1789761061] req@ffff8dc0107afb80 x1876699512813568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761061.661465:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699512813568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761061.661472:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761061.661475:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761061.661478:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512813568:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761061.661485:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512813696:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761061.661506:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761061.661519:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761061/real 1789761061] req@ffff8dc0107ae300 x1876699512813696/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761061.661540:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512813696/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761061.661548:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761061.661552:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761061.661555:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512813696:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761061.661562:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512813824:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761061.661583:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761061.661596:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761061/real 1789761061] req@ffff8dc0107ac700 x1876699512813824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761061.661605:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512813824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761061.661615:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761061.661618:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761061.661620:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512813824:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761062.685051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761063.709332:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761064.733036:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761065.757051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761066.781008:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761066.781019:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1789761066.781020:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761066.781023:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761066.781036:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1041 00000100:00080000:3.0:1789761066.781046:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761066.781053:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761066.781092:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512813952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761066.781126:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761066.781133:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761066.781136:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761066.781142:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1041 00000100:00080000:3.0:1789761066.781147:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761066.781153:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761066.781159:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512813952:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789761066.781165:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761066.781183:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512814080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761066.781197:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761066.781201:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761066.781203:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761066.781208:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1041 00000100:00080000:3.0:1789761066.781212:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761066.781214:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761066.781221:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761066.781228:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761066.781244:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512814208/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761066.781254:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761066/real 1789761066] req@ffff8dc0107ac000 x1876699512813952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761066.781258:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761066.781267:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761066.781270:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761066.781277:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1041 00000100:00100000:1.0:1789761066.781278:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512813952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761066.781284:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761066.781292:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761066.781293:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1789761066.781298:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761066.781303:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512813952:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789761066.781303:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761066.781320:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512814336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761066.781324:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512814080:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761066.781348:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761066.781362:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761066/real 1789761066] req@ffff8dc0107aca80 x1876699512814080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761066.781375:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512814080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761066.781385:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761066.781389:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761066.781392:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512814080:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761066.781400:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512814208:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761066.781419:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761066.781428:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761066/real 1789761066] req@ffff8dc0107ad500 x1876699512814208/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761066.781436:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512814208/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761066.781442:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761066.781444:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761066.781446:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512814208:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761066.781456:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512814336:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761066.781467:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761066.781474:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761066/real 1789761066] req@ffff8dc0107aed80 x1876699512814336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761066.781484:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512814336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761066.781493:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761066.781496:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761066.781499:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512814336:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761067.805077:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761068.829026:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761069.853133:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761070.877288:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761071.901000:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1789761071.901012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761071.901024:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761071.901028:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761071.901039:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1046 00000100:00080000:2.0:1789761071.901048:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761071.901056:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761071.901089:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512814464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761071.901114:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761071.901117:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761071.901118:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761071.901122:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1046 00000100:00080000:2.0:1789761071.901125:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761071.901129:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761071.901137:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761071.901145:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512814464:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761071.901154:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512814592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761071.901167:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761071.901170:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761071.901172:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761071.901177:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1046 00000100:00080000:2.0:1789761071.901183:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761071.901187:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761071.901193:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789761071.901202:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761071.901208:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512814720/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761071.901222:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761071.901225:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761071.901228:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761071.901233:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1046 00000100:00080000:2.0:1789761071.901237:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789761071.901239:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761071/real 1789761071] req@ffff8dc00368f480 x1876699512814464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761071.901244:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761071.901252:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761071.901261:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512814464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761071.901266:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512814848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761071.901276:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761071.901283:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761071.901299:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512814464:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761071.901318:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512814592:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761071.901332:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761071.901341:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761071/real 1789761071] req@ffff8dc00368fb80 x1876699512814592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761071.901353:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512814592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761071.901361:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761071.901364:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761071.901375:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512814592:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761071.901382:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512814720:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761071.901404:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761071.901413:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761071/real 1789761071] req@ffff8dc00368d880 x1876699512814720/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761071.901429:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512814720/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761071.901443:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761071.901446:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761071.901449:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512814720:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761071.901456:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512814848:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761071.901477:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761071.901489:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761071/real 1789761071] req@ffff8dc00368f100 x1876699512814848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761071.901497:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512814848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761071.901503:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761071.901505:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761071.901507:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512814848:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761072.925080:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761073.949040:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761074.973009:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761075.997089:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761077.021031:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761077.021042:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761077.021046:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761077.021059:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1051 00000100:00080000:2.0:1789761077.021072:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761077.021080:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789761077.021095:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789761077.021126:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512814976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761077.021152:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761077.021156:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761077.021157:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761077.021161:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1051 00000100:00080000:2.0:1789761077.021164:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761077.021168:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761077.021179:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761077.021192:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512815104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761077.021204:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761077.021207:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761077.021209:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789761077.021213:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512814976:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789761077.021213:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1051 00000100:00080000:2.0:1789761077.021218:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761077.021228:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761077.021233:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761077.021255:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512815232/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761077.021262:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761077.021266:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761077.021268:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761077.021270:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1051 00000100:00080000:2.0:1789761077.021274:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761077.021278:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761077.021282:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761077.021289:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512815360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761077.021306:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761077.021353:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761077/real 1789761077] req@ffff8dc00368f100 x1876699512814976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761077.021386:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512814976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761077.021394:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761077.021398:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761077.021402:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512814976:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761077.021421:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512815104:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761077.021438:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761077.021447:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761077/real 1789761077] req@ffff8dc00368d880 x1876699512815104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761077.021455:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512815104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761077.021460:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761077.021463:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761077.021465:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512815104:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761077.021469:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512815232:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761077.021480:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761077.021486:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761077/real 1789761077] req@ffff8dc00368fb80 x1876699512815232/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761077.021501:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512815232/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761077.021506:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761077.021509:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761077.021511:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512815232:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761077.021515:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512815360:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761077.021527:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761077.021534:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761077/real 1789761077] req@ffff8dc00368f480 x1876699512815360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761077.021541:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512815360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761077.021546:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761077.021548:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761077.021550:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512815360:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761078.045062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761079.069049:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761080.093059:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761081.117047:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761082.141022:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761082.141031:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761082.141035:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761082.141045:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1056 00000100:00080000:3.0:1789761082.141057:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761082.141064:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761082.141073:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761082.141099:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512815488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761082.141120:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761082.141123:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761082.141125:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761082.141128:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1056 00000100:00080000:3.0:1789761082.141130:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761082.141133:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761082.141140:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761082.141149:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512815616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761082.141158:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761082.141159:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512815488:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761082.141162:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761082.141167:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761082.141171:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1056 00000100:00080000:3.0:1789761082.141173:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761082.141176:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761082.141179:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761082.141188:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512815744/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761082.141196:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761082.141198:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761082.141200:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761082.141202:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1056 00000100:00080000:3.0:1789761082.141204:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761082.141207:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761082.141210:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761082.141217:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512815872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761082.141267:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512815616:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761082.141294:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512815744:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761082.141307:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512815872:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761083.165149:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761084.189075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761085.213223:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00010000:00010000:0.0:1789761085.213265:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761085.213271:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789761085.213289:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789761085.213323:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789761085.213336:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789761085.213345:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1789761085.213357:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761082/real 1789761085] req@ffff8dc0107ad180 x1876699512815616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761085.213384:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512815616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761085.213396:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761085.213402:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761085.213406:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512815616:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761085.213414:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761082/real 1789761085] req@ffff8dc0107af480 x1876699512815744/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761085.213433:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512815744/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761085.213442:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761085.213446:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761085.213450:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512815744:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761085.213456:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761082/real 1789761085] req@ffff8dc0107adf80 x1876699512815872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761085.213469:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512815872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761085.213477:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761085.213481:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761085.213485:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512815872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761085.213492:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761082/real 1789761085] req@ffff8dc0107ac380 x1876699512815488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761085.213504:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512815488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761085.213512:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761085.213515:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761085.213519:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512815488:192.168.203.156@tcp:250:kworker.0 00010000:00010000:3.0:1789761086.237122:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761087.260980:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761087.260990:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761087.260995:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761087.261006:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1061 00000100:00080000:3.0:1789761087.261017:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761087.261024:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789761087.261050:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761087.261062:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512816000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761087.261089:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761087.261093:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761087.261094:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761087.261098:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1061 00000100:00080000:3.0:1789761087.261101:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761087.261105:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761087.261114:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761087.261125:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512816128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761087.261133:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512816000:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761087.261135:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761087.261140:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761087.261141:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761087.261145:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1061 00000100:00080000:3.0:1789761087.261150:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761087.261152:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761087.261155:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761087.261165:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512816256/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761087.261173:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761087.261175:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761087.261176:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761087.261179:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1061 00000100:00080000:3.0:1789761087.261182:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761087.261185:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761087.261187:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761087.261195:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512816384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761087.261206:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761087.261260:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761087/real 1789761087] req@ffff8dc0107af800 x1876699512816000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761087.261280:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512816000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761087.261292:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761087.261297:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761087.261301:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512816000:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761087.261336:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512816128:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761087.261361:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761087.261374:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761087/real 1789761087] req@ffff8dc0107ae680 x1876699512816128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761087.261387:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512816128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761087.261395:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761087.261399:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761087.261402:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512816128:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761087.261408:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512816256:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761087.261427:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761087.261438:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761087/real 1789761087] req@ffff8dc0107aea00 x1876699512816256/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761087.261455:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512816256/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761087.261463:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761087.261466:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761087.261469:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512816256:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761087.261475:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512816384:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761087.261495:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761087.261505:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761087/real 1789761087] req@ffff8dc0107ad880 x1876699512816384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761087.261518:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512816384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761087.261527:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761087.261530:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761087.261533:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512816384:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761088.285123:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761089.308991:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761090.333024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761091.356992:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761092.381000:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1789761092.381007:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761092.381010:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761092.381016:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761092.381027:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1066 00000100:00080000:3.0:1789761092.381037:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761092.381043:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761092.381081:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512816512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761092.381101:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761092.381104:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761092.381106:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761092.381108:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1066 00000100:00080000:3.0:1789761092.381112:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761092.381116:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761092.381124:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761092.381132:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512816512:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761092.381137:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512816640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761092.381145:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761092.381147:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761092.381149:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761092.381151:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1066 00000100:00080000:3.0:1789761092.381153:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761092.381155:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761092.381158:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761092.381168:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512816768/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761092.381174:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761092.381177:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761092.381179:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761092.381181:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1066 00000100:00080000:3.0:1789761092.381183:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761092.381186:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1789761092.381190:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789761092.381190:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761092.381200:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512816896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761092.381234:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761092/real 1789761092] req@ffff8dc0107af100 x1876699512816512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761092.381249:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512816512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761092.381263:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761092.381267:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761092.381271:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512816512:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761092.381289:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512816640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761092.381304:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761092.381312:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761092/real 1789761092] req@ffff8dc0107ace00 x1876699512816640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761092.381320:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512816640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761092.381325:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761092.381327:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761092.381330:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512816640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761092.381334:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512816768:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761092.381345:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761092.381351:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761092/real 1789761092] req@ffff8dc0107ad880 x1876699512816768/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761092.381359:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512816768/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761092.381366:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761092.381368:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761092.381371:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512816768:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761092.381377:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512816896:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761092.381390:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761092.381396:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761092/real 1789761092] req@ffff8dc0107aea00 x1876699512816896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761092.381403:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512816896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761092.381408:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761092.381410:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761092.381413:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512816896:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761093.405019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761094.428992:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761095.453062:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761096.477006:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761097.501034:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761097.501044:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761097.501048:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761097.501059:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1072 00010000:00010000:0.0:1789761097.501085:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761097.501087:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761097.501097:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761097.501155:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512817024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761097.501188:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761097.501193:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761097.501196:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761097.501202:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1072 00000100:00080000:2.0:1789761097.501208:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761097.501214:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761097.501226:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761097.501236:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512817024:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761097.501247:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512817152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761097.501260:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761097.501274:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761097.501277:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761097.501283:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1072 00000100:00080000:2.0:1789761097.501293:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761097.501300:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1789761097.501304:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:2.0:1789761097.501312:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761097.501334:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512817280/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761097.501363:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761097/real 1789761097] req@ffff8dc00368c000 x1876699512817024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761097.501366:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761097.501376:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761097.501381:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761097.501388:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1072 00000100:00100000:1.0:1789761097.501391:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512817024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761097.501395:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761097.501406:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761097.501407:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761097.501418:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789761097.501422:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761097.501428:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512817024:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761097.501435:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512817408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761097.501450:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512817152:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761097.501473:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761097.501485:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761097/real 1789761097] req@ffff8dc00368ca80 x1876699512817152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761097.501494:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512817152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761097.501501:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761097.501503:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761097.501506:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512817152:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761097.501510:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512817280:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761097.501522:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761097.501529:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761097/real 1789761097] req@ffff8dc00368ce00 x1876699512817280/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761097.501538:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512817280/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761097.501544:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761097.501546:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761097.501548:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512817280:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761097.501557:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512817408:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761097.501569:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761097.501576:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761097/real 1789761097] req@ffff8dc00368ea00 x1876699512817408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761097.501584:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512817408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761097.501590:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761097.501592:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761097.501595:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512817408:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761098.525008:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761099.549007:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761100.573028:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761101.597027:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761102.621016:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761102.621025:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761102.621030:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1789761102.621039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761102.621042:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1077 00000100:00080000:3.0:1789761102.621052:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761102.621058:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761102.621089:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512817536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761102.621110:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761102.621114:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761102.621116:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761102.621121:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1077 00000100:00080000:3.0:1789761102.621125:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761102.621129:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761102.621139:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761102.621144:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512817536:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761102.621151:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512817664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761102.621161:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761102.621164:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761102.621165:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761102.621169:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1077 00000100:00080000:3.0:1789761102.621172:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761102.621175:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761102.621180:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761102.621190:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512817792/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761102.621199:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761102.621202:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1789761102.621203:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761102.621203:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761102.621208:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1077 00000100:00080000:3.0:1789761102.621211:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761102.621214:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761102.621218:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761102.621227:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512817920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761102.621248:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761102/real 1789761102] req@ffff8dc0107ae680 x1876699512817536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761102.621268:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512817536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761102.621278:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761102.621281:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761102.621284:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512817536:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761102.621302:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512817664:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761102.621329:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761102.621339:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761102/real 1789761102] req@ffff8dc0107af800 x1876699512817664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761102.621347:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512817664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761102.621352:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761102.621355:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761102.621357:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512817664:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761102.621361:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512817792:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761102.621373:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761102.621379:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761102/real 1789761102] req@ffff8dc0107ac380 x1876699512817792/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761102.621386:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512817792/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761102.621392:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761102.621394:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761102.621396:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512817792:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761102.621399:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512817920:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761102.621409:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761102.621414:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761102/real 1789761102] req@ffff8dc0107adf80 x1876699512817920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761102.621421:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512817920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761102.621426:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761102.621428:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761102.621430:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512817920:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761103.644995:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761104.669020:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761105.693092:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761106.717017:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761107.741004:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761107.741015:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761107.741020:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1789761107.741022:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761107.741032:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1082 00000100:00080000:1.0:1789761107.741042:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761107.741050:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761107.741089:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512818048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761107.741115:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761107.741119:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761107.741121:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761107.741126:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1082 00000100:00080000:1.0:1789761107.741130:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761107.741134:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761107.741142:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761107.741152:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512818048:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761107.741156:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512818176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761107.741169:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761107.741173:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761107.741175:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761107.741180:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1082 00000100:00080000:1.0:1789761107.741184:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761107.741188:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761107.741194:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1789761107.741206:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761107.741210:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512818304/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761107.741230:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761107.741234:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761107.741236:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789761107.741241:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761107/real 1789761107] req@ffff8dc005c97800 x1876699512818048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761107.741245:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1082 00000100:00080000:1.0:1789761107.741254:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761107.741261:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761107.741263:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512818048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789761107.741271:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789761107.741275:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761107.741281:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761107.741288:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512818048:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761107.741290:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512818432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761107.741318:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512818176:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761107.741346:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761107.741361:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761107/real 1789761107] req@ffff8dc005c95f80 x1876699512818176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761107.741376:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512818176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761107.741385:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761107.741388:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761107.741392:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512818176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761107.741399:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512818304:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761107.741422:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761107.741435:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761107/real 1789761107] req@ffff8dc005c95500 x1876699512818304/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761107.741448:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512818304/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761107.741457:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761107.741461:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761107.741465:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512818304:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761107.741472:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512818432:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761107.741494:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761107.741506:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761107/real 1789761107] req@ffff8dc005c96a00 x1876699512818432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761107.741527:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512818432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761107.741536:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761107.741540:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761107.741543:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512818432:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761108.765105:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761109.789097:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761110.813037:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761111.837086:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761112.861005:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761112.861014:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761112.861017:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789761112.861026:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761112.861027:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1087 00000100:00080000:1.0:1789761112.861039:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761112.861046:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761112.861080:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512818560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761112.861107:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761112.861112:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761112.861114:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761112.861118:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1087 00000100:00080000:1.0:1789761112.861122:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761112.861128:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761112.861139:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761112.861155:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512818688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761112.861157:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512818560:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761112.861175:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761112.861179:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761112.861182:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761112.861187:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1087 00000100:00080000:1.0:1789761112.861192:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761112.861196:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761112.861202:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761112.861215:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512818816/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761112.861222:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761112.861230:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761112.861233:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761112.861234:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761112.861238:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1087 00000100:00080000:1.0:1789761112.861241:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761112.861245:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761112.861248:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761112.861257:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512818944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761112.861272:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761112/real 1789761112] req@ffff8dc005c94700 x1876699512818560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761112.861288:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512818560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761112.861298:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761112.861301:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761112.861304:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512818560:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761112.861322:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512818688:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761112.861337:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761112.861346:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761112/real 1789761112] req@ffff8dc005c94380 x1876699512818688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761112.861354:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512818688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761112.861364:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761112.861367:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761112.861369:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512818688:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761112.861373:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512818816:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761112.861384:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761112.861390:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761112/real 1789761112] req@ffff8dc005c97b80 x1876699512818816/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761112.861398:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512818816/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761112.861403:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761112.861405:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761112.861407:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512818816:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761112.861411:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512818944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761112.861420:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761112.861426:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761112/real 1789761112] req@ffff8dc005c96300 x1876699512818944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761112.861433:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512818944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761112.861438:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761112.861440:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761112.861443:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512818944:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761113.885158:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761114.909092:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761115.932998:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761116.957133:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761117.980977:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761117.980996:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761117.981002:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761117.981016:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1092 00010000:00010000:2.0:1789761117.981021:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761117.981027:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761117.981035:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761117.981076:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512819072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761117.981100:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761117.981103:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761117.981105:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761117.981107:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1092 00000100:00080000:1.0:1789761117.981110:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761117.981112:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761117.981121:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761117.981131:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512819200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761117.981138:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761117.981141:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761117.981143:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761117.981146:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1092 00000100:00080000:1.0:1789761117.981148:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761117.981150:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761117.981153:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761117.981162:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512819328/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761117.981171:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761117.981176:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761117.981177:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789761117.981178:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512819072:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761117.981184:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1092 00000100:00080000:1.0:1789761117.981192:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761117.981196:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761117.981200:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761117.981210:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512819456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761117.981255:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761117.981301:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761117/real 1789761117] req@ffff8dc005c94e00 x1876699512819072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761117.981315:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512819072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761117.981323:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761117.981326:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761117.981329:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512819072:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761117.981337:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512819200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761117.981356:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761117.981364:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761117/real 1789761117] req@ffff8dc005c97480 x1876699512819200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761117.981372:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512819200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761117.981378:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761117.981380:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761117.981382:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512819200:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761117.981407:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512819328:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761117.981420:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761117.981428:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761117/real 1789761117] req@ffff8dc005c96680 x1876699512819328/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761117.981435:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512819328/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761117.981441:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761117.981443:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761117.981445:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512819328:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761117.981449:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512819456:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761117.981459:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761117.981465:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761117/real 1789761117] req@ffff8dc005c95880 x1876699512819456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761117.981472:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512819456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761117.981477:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761117.981479:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761117.981481:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512819456:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761119.005041:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761120.029091:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761121.053110:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761122.077051:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761123.101063:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761123.101074:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761123.101079:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1789761123.101086:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761123.101093:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1097 00000100:00080000:1.0:1789761123.101106:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761123.101115:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761123.101158:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96d80 x1876699512819584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761123.101180:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761123.101183:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761123.101184:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761123.101187:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1097 00000100:00080000:1.0:1789761123.101190:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761123.101194:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761123.101204:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761123.101219:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512819712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761123.101220:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512819584:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761123.101234:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761123.101236:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761123.101237:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761123.101240:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1097 00000100:00080000:1.0:1789761123.101243:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761123.101247:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761123.101252:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761123.101266:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512819840/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761123.101277:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761123.101282:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761123.101285:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761123.101289:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1097 00000800:00000100:3.0:1789761123.101294:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761123.101295:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761123.101303:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761123.101310:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761123.101325:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512819968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761123.101346:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761123/real 1789761123] req@ffff8dc005c96d80 x1876699512819584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761123.101373:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96d80 x1876699512819584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761123.101383:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761123.101387:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761123.101390:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512819584:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761123.101412:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512819712:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761123.101429:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761123.101439:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761123/real 1789761123] req@ffff8dc005c95880 x1876699512819712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761123.101447:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512819712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761123.101453:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761123.101456:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761123.101458:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512819712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761123.101462:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512819840:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761123.101475:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761123.101487:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761123/real 1789761123] req@ffff8dc005c96680 x1876699512819840/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761123.101496:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512819840/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761123.101506:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761123.101509:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761123.101511:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512819840:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761123.101514:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512819968:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761123.101526:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761123.101532:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761123/real 1789761123] req@ffff8dc005c97480 x1876699512819968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761123.101540:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512819968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761123.101545:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761123.101547:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761123.101549:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512819968:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761124.125067:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761125.149017:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761126.173056:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761127.197044:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761128.221116:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761128.221219:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761128.221225:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761128.221228:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761128.221236:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1102 00000100:00080000:2.0:1789761128.221245:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761128.221250:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761128.221278:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512820096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761128.221297:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761128.221300:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761128.221302:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761128.221305:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1102 00000100:00080000:2.0:1789761128.221308:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761128.221312:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761128.221320:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761128.221330:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512820224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761128.221338:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761128.221340:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761128.221342:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761128.221344:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1102 00000100:00080000:2.0:1789761128.221346:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761128.221352:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761128.221352:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512820096:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789761128.221357:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761128.221368:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512820352/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761128.221375:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761128.221378:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761128.221379:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761128.221382:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1102 00000100:00080000:2.0:1789761128.221384:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761128.221386:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761128.221390:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761128.221397:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512820480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761128.221555:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761128.221643:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761128/real 1789761128] req@ffff8dc00368f800 x1876699512820096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761128.221664:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512820096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761128.221675:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761128.221679:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761128.221684:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512820096:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761128.221692:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512820224:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761128.221718:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761128.221732:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761128/real 1789761128] req@ffff8dc00368ed80 x1876699512820224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761128.221746:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512820224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761128.221756:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761128.221760:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761128.221763:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512820224:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761128.221785:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512820352:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761128.221807:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761128.221820:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761128/real 1789761128] req@ffff8dc00368e300 x1876699512820352/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761128.221832:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512820352/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761128.221842:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761128.221845:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761128.221849:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512820352:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761128.221856:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512820480:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761128.221883:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761128.221897:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761128/real 1789761128] req@ffff8dc00368df80 x1876699512820480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761128.221940:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512820480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761128.221949:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761128.221953:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761128.221958:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512820480:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761129.245028:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761130.269077:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761131.293073:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761132.317063:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761133.340993:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761133.341004:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761133.341008:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761133.341020:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1107 00000100:00080000:2.0:1789761133.341032:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761133.341039:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789761133.341053:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789761133.341073:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512820608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761133.341103:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761133.341105:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761133.341107:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761133.341110:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1107 00000100:00080000:2.0:1789761133.341112:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761133.341116:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761133.341124:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761133.341133:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512820736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761133.341138:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512820608:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789761133.341144:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761133.341148:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761133.341150:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761133.341152:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1107 00000100:00080000:2.0:1789761133.341154:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761133.341156:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761133.341166:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761133.341177:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512820864/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761133.341185:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761133.341187:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761133.341188:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761133.341191:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1107 00000100:00080000:2.0:1789761133.341196:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761133.341198:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761133.341202:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789761133.341207:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761133.341214:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512820992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761133.341244:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761133/real 1789761133] req@ffff8dc00368e680 x1876699512820608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761133.341258:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512820608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761133.341266:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761133.341268:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761133.341271:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512820608:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761133.341288:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512820736:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761133.341303:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761133.341311:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761133/real 1789761133] req@ffff8dc00368d180 x1876699512820736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761133.341324:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512820736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761133.341330:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761133.341332:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761133.341334:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512820736:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761133.341338:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512820864:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761133.341348:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761133.341354:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761133/real 1789761133] req@ffff8dc00368c700 x1876699512820864/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761133.341362:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512820864/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761133.341367:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761133.341368:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761133.341370:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512820864:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761133.341374:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512820992:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761133.341383:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761133.341389:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761133/real 1789761133] req@ffff8dc00368dc00 x1876699512820992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761133.341396:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512820992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761133.341405:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761133.341407:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761133.341409:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512820992:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761134.365044:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761135.389035:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761136.413090:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761137.437032:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761138.461006:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761138.461014:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761138.461018:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761138.461027:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1113 00000100:00080000:1.0:1789761138.461036:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761138.461042:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789761138.461047:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761138.461076:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512821120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761138.461095:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761138.461098:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761138.461099:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761138.461102:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1113 00000100:00080000:1.0:1789761138.461104:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761138.461107:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761138.461114:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761138.461124:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512821120:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761138.461125:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512821248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761138.461140:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761138.461142:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761138.461143:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761138.461145:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1113 00000100:00080000:1.0:1789761138.461147:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761138.461149:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761138.461152:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761138.461159:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512821376/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761138.461171:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761138.461174:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761138.461175:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761138.461177:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1113 00000100:00080000:1.0:1789761138.461179:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761138.461182:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761138.461185:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789761138.461193:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761138.461194:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512821504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761138.461238:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761138/real 1789761138] req@ffff8dc005c94e00 x1876699512821120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761138.461253:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512821120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761138.461262:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761138.461265:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761138.461268:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512821120:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761138.461287:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512821248:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761138.461304:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761138.461312:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761138/real 1789761138] req@ffff8dc005c96300 x1876699512821248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761138.461320:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512821248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761138.461330:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761138.461333:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761138.461335:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512821248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761138.461339:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512821376:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761138.461350:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761138.461356:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761138/real 1789761138] req@ffff8dc005c97b80 x1876699512821376/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761138.461364:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512821376/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761138.461369:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761138.461371:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761138.461373:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512821376:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761138.461377:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512821504:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761138.461386:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761138.461395:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761138/real 1789761138] req@ffff8dc005c94380 x1876699512821504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761138.461407:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512821504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761138.461415:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761138.461419:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761138.461423:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512821504:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761139.485070:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761140.509022:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761141.533056:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761142.557157:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761143.581001:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761143.581010:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761143.581014:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761143.581023:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1118 00010000:00010000:0.0:1789761143.581025:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761143.581031:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761143.581038:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761143.581069:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512821632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761143.581089:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761143.581092:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761143.581094:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761143.581096:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1118 00000100:00080000:3.0:1789761143.581099:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761143.581104:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761143.581113:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761143.581126:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512821632:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761143.581130:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512821760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761143.581144:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761143.581149:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761143.581151:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761143.581156:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1118 00000100:00080000:3.0:1789761143.581161:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761143.581164:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761143.581168:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761143.581180:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512821888/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761143.581183:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761143.581194:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761143.581196:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761143.581198:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761143.581202:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1118 00000100:00080000:3.0:1789761143.581205:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761143.581207:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761143.581211:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761143.581225:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761143/real 1789761143] req@ffff8dc0107af480 x1876699512821632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761143.581225:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512822016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761143.581258:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512821632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761143.581268:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761143.581271:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761143.581273:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512821632:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761143.581292:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512821760:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761143.581306:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761143.581314:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761143/real 1789761143] req@ffff8dc0107ad180 x1876699512821760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761143.581323:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512821760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761143.581328:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761143.581330:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761143.581332:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512821760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761143.581336:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512821888:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761143.581347:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761143.581352:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761143/real 1789761143] req@ffff8dc0107aed80 x1876699512821888/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761143.581360:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512821888/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761143.581365:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761143.581366:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761143.581368:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512821888:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761143.581371:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512822016:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761143.581381:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761143.581387:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761143/real 1789761143] req@ffff8dc0107ad500 x1876699512822016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761143.581394:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512822016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761143.581403:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761143.581405:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761143.581407:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512822016:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761144.605030:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761145.629127:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761146.653020:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761147.677075:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761148.700971:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761148.700980:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761148.700983:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761148.700992:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1123 00000100:00080000:3.0:1789761148.701003:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761148.701009:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761148.701043:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512822144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1789761148.701051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761148.701065:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761148.701069:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761148.701071:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761148.701074:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1123 00000100:00080000:3.0:1789761148.701077:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761148.701080:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761148.701087:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761148.701097:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512822144:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761148.701099:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512822272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761148.701112:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761148.701115:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761148.701116:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761148.701119:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1123 00000100:00080000:3.0:1789761148.701121:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761148.701123:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761148.701126:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761148.701134:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512822400/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761148.701141:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761148.701143:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761148.701145:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761148.701148:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1123 00000100:00080000:3.0:1789761148.701151:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761148.701153:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761148.701156:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761148.701164:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512822528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761148.701196:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512822272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761148.701222:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512822400:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761148.701235:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512822528:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761149.724996:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761150.749030:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761151.773130:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789761151.773199:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1789761151.773235:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789761151.773249:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761151.773282:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761151.773288:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761151.773303:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1789761151.773327:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761148/real 1789761151] req@ffff8dc0107ac000 x1876699512822272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761151.773370:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512822272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761151.773386:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761151.773392:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761151.773398:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512822272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761151.773408:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761148/real 1789761151] req@ffff8dc0107ac700 x1876699512822400/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761151.773423:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512822400/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761151.773432:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761151.773436:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761151.773440:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512822400:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761151.773448:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761148/real 1789761151] req@ffff8dc0107ae300 x1876699512822528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761151.773472:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512822528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761151.773481:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761151.773485:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761151.773489:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512822528:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761151.773497:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761148/real 1789761151] req@ffff8dc0107aca80 x1876699512822144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761151.773511:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512822144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761151.773520:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761151.773524:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761151.773528:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512822144:192.168.203.156@tcp:250:kworker.0 00010000:00010000:0.0:1789761152.797022:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761153.820994:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761153.821002:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761153.821006:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761153.821014:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1128 00000100:00080000:3.0:1789761153.821022:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761153.821028:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761153.821061:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699512822656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761153.821084:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761153.821087:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761153.821090:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761153.821093:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1128 00000100:00080000:3.0:1789761153.821096:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761153.821099:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761153.821107:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00010000:00010000:1.0:1789761153.821111:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761153.821119:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699512822784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761153.821130:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512822656:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761153.821134:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761153.821141:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761153.821142:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761153.821146:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1128 00000100:00080000:3.0:1789761153.821148:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761153.821155:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761153.821158:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761153.821169:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512822912/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761153.821176:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761153.821179:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761153.821180:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761153.821183:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1128 00000100:00080000:3.0:1789761153.821185:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761153.821187:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761153.821191:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761153.821203:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512823040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761153.821206:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761153.821270:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761153/real 1789761153] req@ffff8dc0107afb80 x1876699512822656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761153.821291:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699512822656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761153.821302:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761153.821307:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761153.821311:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512822656:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761153.821335:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512822784:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761153.821377:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761153.821392:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761153/real 1789761153] req@ffff8dc0107adc00 x1876699512822784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761153.821404:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699512822784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761153.821413:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761153.821417:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761153.821420:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512822784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761153.821427:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512822912:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761153.821450:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761153.821463:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761153/real 1789761153] req@ffff8dc0107aca80 x1876699512822912/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761153.821480:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512822912/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761153.821487:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761153.821490:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761153.821495:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512822912:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761153.821500:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512823040:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761153.821520:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761153.821531:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761153/real 1789761153] req@ffff8dc0107ae300 x1876699512823040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761153.821543:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512823040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761153.821551:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761153.821554:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761153.821557:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512823040:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761154.845061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761155.869031:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761156.893047:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761157.917096:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761158.940989:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761158.940997:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761158.941000:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761158.941010:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1133 00010000:00010000:0.0:1789761158.941015:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761158.941019:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761158.941025:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761158.941057:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512823168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761158.941080:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761158.941083:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761158.941084:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761158.941087:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1133 00000100:00080000:3.0:1789761158.941089:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761158.941091:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761158.941098:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761158.941107:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512823296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761158.941115:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761158.941118:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512823168:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761158.941119:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761158.941123:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761158.941130:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1133 00000100:00080000:3.0:1789761158.941132:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761158.941134:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761158.941138:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761158.941147:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512823424/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761158.941154:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761158.941157:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761158.941159:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761158.941162:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1133 00000100:00080000:3.0:1789761158.941165:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761158.941167:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761158.941170:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761158.941178:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512823552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761158.941181:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761158.941249:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761158/real 1789761158] req@ffff8dc0107ac700 x1876699512823168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761158.941267:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512823168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761158.941276:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761158.941279:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761158.941281:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512823168:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761158.941314:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512823296:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761158.941334:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761158.941365:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761158/real 1789761158] req@ffff8dc0107ac000 x1876699512823296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761158.941376:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512823296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761158.941384:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761158.941387:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761158.941390:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512823296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761158.941396:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512823424:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761158.941412:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761158.941421:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761158/real 1789761158] req@ffff8dc0107ad500 x1876699512823424/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761158.941432:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512823424/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761158.941445:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761158.941447:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761158.941450:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512823424:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761158.941455:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512823552:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761158.941484:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761158.941491:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761158/real 1789761158] req@ffff8dc0107aed80 x1876699512823552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761158.941498:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512823552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761158.941503:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761158.941505:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761158.941507:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512823552:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761159.965054:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761160.989060:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761162.013029:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761163.037078:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761164.061084:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761164.061092:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761164.061095:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761164.061103:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1138 00000100:00080000:0.0:1789761164.061116:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1789761164.061123:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761164.061126:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761164.061163:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149880 x1876699512823680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761164.061187:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761164.061190:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761164.061191:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761164.061195:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1138 00000100:00080000:0.0:1789761164.061199:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761164.061202:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761164.061211:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761164.061231:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b800 x1876699512823808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761164.061236:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512823680:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789761164.061276:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761164.061279:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761164.061281:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761164.061289:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1138 00000100:00080000:0.0:1789761164.061292:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761164.061296:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761164.061305:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761164.061321:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149f80 x1876699512823936/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761164.061328:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761164.061331:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761164.061333:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761164.061337:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1138 00000100:00080000:0.0:1789761164.061340:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761164.061343:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761164.061347:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789761164.061354:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761164.061364:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149180 x1876699512824064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761164.061415:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761164/real 1789761164] req@ffff8dc005149880 x1876699512823680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761164.061533:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149880 x1876699512823680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761164.061664:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761164.061669:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761164.061674:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512823680:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761164.061738:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512823808:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761164.061768:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761164.061792:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761164/real 1789761164] req@ffff8dc00514b800 x1876699512823808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761164.061808:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b800 x1876699512823808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761164.061818:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761164.061822:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761164.061826:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512823808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761164.061834:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512823936:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761164.061853:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761164.061863:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761164/real 1789761164] req@ffff8dc005149f80 x1876699512823936/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761164.061871:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149f80 x1876699512823936/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761164.061877:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761164.061880:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761164.061882:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512823936:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761164.061886:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512824064:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761164.061897:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761164.061936:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761164/real 1789761164] req@ffff8dc005149180 x1876699512824064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761164.061946:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149180 x1876699512824064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761164.061957:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761164.061961:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761164.061964:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512824064:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761165.085090:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761166.109044:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761167.133054:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761168.157045:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761169.180998:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761169.181013:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761169.181019:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761169.181034:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1143 00000100:00080000:2.0:1789761169.181047:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1789761169.181052:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761169.181057:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761169.181120:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512824192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761169.181147:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761169.181151:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761169.181154:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761169.181159:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1143 00000100:00080000:2.0:1789761169.181164:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761169.181169:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761169.181180:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761169.181187:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512824192:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761169.181196:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512824320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761169.181208:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761169.181212:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761169.181213:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761169.181217:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1143 00000100:00080000:2.0:1789761169.181220:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761169.181234:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761169.181240:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761169.181252:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512824448/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761169.181265:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761169.181268:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761169.181270:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761169.181273:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1143 00000100:00080000:2.0:1789761169.181276:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789761169.181280:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789761169.181283:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761169.181289:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761169.181296:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512824576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761169.181338:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761169/real 1789761169] req@ffff8dc00368dc00 x1876699512824192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761169.181355:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512824192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761169.181363:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761169.181366:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761169.181370:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512824192:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761169.181403:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512824320:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761169.181421:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761169.181430:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761169/real 1789761169] req@ffff8dc00368c700 x1876699512824320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761169.181439:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512824320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761169.181462:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761169.181465:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761169.181470:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512824320:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761169.181476:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512824448:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761169.181498:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761169.181511:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761169/real 1789761169] req@ffff8dc00368d180 x1876699512824448/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761169.181525:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512824448/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761169.181535:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761169.181538:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761169.181542:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512824448:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761169.181547:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512824576:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761169.181563:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761169.181571:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761169/real 1789761169] req@ffff8dc00368e680 x1876699512824576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761169.181579:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512824576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761169.181585:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761169.181587:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761169.181590:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512824576:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761170.205071:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761171.229079:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761172.253030:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761173.277041:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761174.301004:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761174.301013:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761174.301017:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789761174.301020:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761174.301026:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1148 00000100:00080000:1.0:1789761174.301034:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761174.301041:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761174.301074:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512824704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761174.301095:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761174.301098:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761174.301099:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761174.301102:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1148 00000100:00080000:1.0:1789761174.301105:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761174.301107:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761174.301115:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761174.301124:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512824832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761174.301132:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761174.301136:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761174.301138:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1789761174.301138:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512824704:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761174.301144:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1148 00000100:00080000:1.0:1789761174.301148:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761174.301151:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761174.301162:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761174.301171:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512824960/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761174.301178:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761174.301181:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761174.301183:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761174.301198:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1148 00000100:00080000:1.0:1789761174.301201:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761174.301204:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761174.301208:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761174.301218:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512825088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761174.301229:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761174.301275:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761174/real 1789761174] req@ffff8dc005c94700 x1876699512824704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761174.301290:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512824704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761174.301299:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761174.301302:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761174.301305:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512824704:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761174.301322:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512824832:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761174.301341:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761174.301354:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761174/real 1789761174] req@ffff8dc005c96a00 x1876699512824832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761174.301367:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512824832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761174.301376:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761174.301380:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761174.301383:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512824832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761174.301390:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512824960:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761174.301413:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761174.301422:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761174/real 1789761174] req@ffff8dc005c95500 x1876699512824960/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761174.301430:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512824960/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761174.301436:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761174.301438:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761174.301440:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512824960:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761174.301444:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512825088:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761174.301456:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761174.301464:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761174/real 1789761174] req@ffff8dc005c95f80 x1876699512825088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761174.301476:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512825088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761174.301484:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761174.301488:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761174.301492:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512825088:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761175.325001:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761176.349049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761177.373009:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761178.397047:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761179.421003:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761179.421011:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761179.421015:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761179.421023:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1154 00000100:00080000:1.0:1789761179.421031:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761179.421037:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761179.421047:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761179.421071:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512825216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761179.421097:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761179.421102:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761179.421104:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761179.421109:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1154 00000100:00080000:1.0:1789761179.421113:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761179.421117:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761179.421125:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761179.421129:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512825216:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761179.421141:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512825344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761179.421163:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761179.421166:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761179.421168:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761179.421171:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1154 00000100:00080000:1.0:1789761179.421174:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761179.421177:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761179.421181:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761179.421191:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512825472/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761179.421199:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789761179.421201:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761179.421203:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761179.421205:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761179.421209:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1154 00000100:00080000:1.0:1789761179.421212:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761179.421216:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761179.421219:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761179.421228:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512825600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761179.421240:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761179/real 1789761179] req@ffff8dc005c97800 x1876699512825216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761179.421255:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512825216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761179.421263:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761179.421265:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761179.421268:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512825216:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761179.421286:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512825344:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761179.421301:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761179.421309:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761179/real 1789761179] req@ffff8dc005c94a80 x1876699512825344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761179.421325:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512825344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761179.421331:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761179.421334:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761179.421336:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512825344:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761179.421340:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512825472:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761179.421353:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761179.421360:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761179/real 1789761179] req@ffff8dc005c95180 x1876699512825472/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761179.421367:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512825472/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761179.421377:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761179.421379:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761179.421382:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512825472:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761179.421385:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512825600:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761179.421395:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761179.421401:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761179/real 1789761179] req@ffff8dc005c95c00 x1876699512825600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761179.421408:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512825600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761179.421413:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761179.421415:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761179.421417:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512825600:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761180.445050:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761181.469095:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761182.493102:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761183.517058:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761184.541034:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761184.541044:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761184.541049:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761184.541061:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1159 00000100:00080000:1.0:1789761184.541074:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761184.541083:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789761184.541090:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761184.541140:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512825728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761184.541168:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761184.541172:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761184.541174:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761184.541178:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1159 00000100:00080000:1.0:1789761184.541183:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761184.541189:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761184.541203:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761184.541209:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512825728:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761184.541225:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512825856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761184.541238:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761184.541241:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761184.541242:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761184.541247:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1159 00000100:00080000:1.0:1789761184.541251:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761184.541254:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761184.541260:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761184.541268:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512825984/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761184.541276:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789761184.541288:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761184.541292:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761184.541295:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761184.541299:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1159 00000100:00080000:1.0:1789761184.541303:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761184.541307:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761184.541311:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761184.541322:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512826112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761184.541351:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761184/real 1789761184] req@ffff8dc005c97100 x1876699512825728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761184.541378:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512825728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761184.541390:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761184.541394:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761184.541398:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512825728:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761184.541419:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512825856:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761184.541439:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761184.541450:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761184/real 1789761184] req@ffff8dc005c95c00 x1876699512825856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761184.541458:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512825856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761184.541464:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761184.541466:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761184.541468:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512825856:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761184.541472:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512825984:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761184.541483:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761184.541490:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761184/real 1789761184] req@ffff8dc005c95180 x1876699512825984/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761184.541497:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512825984/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761184.541502:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761184.541505:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761184.541507:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512825984:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761184.541510:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512826112:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761184.541520:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761184.541527:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761184/real 1789761184] req@ffff8dc005c94a80 x1876699512826112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761184.541538:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512826112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761184.541547:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761184.541550:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761184.541554:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512826112:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761185.565420:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761186.588999:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761187.613033:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761188.637095:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761189.661008:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761189.661018:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761189.661022:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761189.661031:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1164 00010000:00010000:0.0:1789761189.661037:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761189.661040:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761189.661049:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761189.661097:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512826240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761189.661118:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761189.661121:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761189.661123:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761189.661128:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1164 00000100:00080000:1.0:1789761189.661132:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761189.661138:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761189.661148:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761189.661154:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512826240:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761189.661177:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512826368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761189.661191:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761189.661195:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761189.661197:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761189.661202:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1164 00000100:00080000:1.0:1789761189.661206:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789761189.661209:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761189.661215:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761189.661222:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761189.661235:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512826496/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761189.661244:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789761189.661246:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761189/real 1789761189] req@ffff8dc005c97800 x1876699512826240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761189.661247:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761189.661250:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761189.661258:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1164 00000100:00080000:1.0:1789761189.661264:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761189.661267:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761189.661271:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512826240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789761189.661273:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761189.661286:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512826624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761189.661286:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761189.661295:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761189.661303:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512826240:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761189.661325:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512826368:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761189.661340:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761189.661349:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761189/real 1789761189] req@ffff8dc005c95f80 x1876699512826368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761189.661357:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512826368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761189.661364:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761189.661366:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761189.661368:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512826368:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761189.661372:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512826496:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761189.661383:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761189.661389:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761189/real 1789761189] req@ffff8dc005c95500 x1876699512826496/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761189.661396:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512826496/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761189.661402:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761189.661404:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761189.661406:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512826496:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761189.661409:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512826624:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761189.661419:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761189.661425:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761189/real 1789761189] req@ffff8dc005c96a00 x1876699512826624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761189.661432:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512826624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761189.661437:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761189.661439:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761189.661441:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512826624:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761190.685044:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761191.709038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761192.733037:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761193.757061:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761194.781029:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761194.781059:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761194.781065:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761194.781079:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1169 00000100:00080000:1.0:1789761194.781102:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761194.781112:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761194.781131:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761194.781169:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512826752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761194.781196:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761194.781199:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761194.781205:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761194.781209:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1169 00000100:00080000:1.0:1789761194.781214:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761194.781218:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761194.781229:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761194.781238:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512826752:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761194.781244:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512826880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761194.781255:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761194.781258:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761194.781261:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761194.781264:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1169 00000100:00080000:1.0:1789761194.781268:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761194.781271:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761194.781277:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761194.781286:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512827008/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761194.781295:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761194.781299:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761194.781302:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789761194.781304:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761194.781311:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1169 00000100:00080000:1.0:1789761194.781316:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761194.781319:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761194.781324:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761194.781334:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512827136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761194.781362:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761194/real 1789761194] req@ffff8dc005c94700 x1876699512826752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761194.781383:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512826752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761194.781393:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761194.781396:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761194.781399:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512826752:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761194.781420:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512826880:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761194.781436:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761194.781444:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761194/real 1789761194] req@ffff8dc005c94380 x1876699512826880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761194.781452:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512826880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761194.781458:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761194.781460:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761194.781462:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512826880:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761194.781466:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512827008:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761194.781479:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761194.781485:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761194/real 1789761194] req@ffff8dc005c97b80 x1876699512827008/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761194.781493:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512827008/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761194.781498:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761194.781500:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761194.781502:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512827008:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761194.781506:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512827136:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761194.781519:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761194.781528:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761194/real 1789761194] req@ffff8dc005c96300 x1876699512827136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761194.781540:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512827136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761194.781549:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761194.781552:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761194.781556:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512827136:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761195.805008:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761196.829117:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761197.853071:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761198.877019:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761199.901022:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761199.901036:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761199.901042:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761199.901055:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1174 00000100:00080000:1.0:1789761199.901066:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761199.901074:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761199.901116:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512827264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761199.901141:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761199.901145:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761199.901146:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761199.901149:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1174 00000100:00080000:1.0:1789761199.901152:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761199.901155:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761199.901164:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761199.901176:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512827392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761199.901186:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789761199.901187:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512827264:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761199.901189:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761199.901193:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761199.901201:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1174 00000100:00080000:1.0:1789761199.901204:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761199.901207:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761199.901212:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761199.901223:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512827520/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761199.901232:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761199.901235:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761199.901236:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761199.901240:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1174 00000100:00080000:1.0:1789761199.901242:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761199.901246:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761199.901249:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761199.901259:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512827648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1789761199.901285:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:3.0:1789761199.901285:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761199.901410:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761199/real 1789761199] req@ffff8dc005c94e00 x1876699512827264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761199.901432:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512827264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761199.901441:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761199.901444:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761199.901448:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512827264:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761199.901469:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512827392:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761199.901488:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761199.901496:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761199/real 1789761199] req@ffff8dc005c97480 x1876699512827392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761199.901505:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512827392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761199.901619:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761199.901622:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761199.901625:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512827392:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761199.901630:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512827520:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761199.901650:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761199.901659:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761199/real 1789761199] req@ffff8dc005c96680 x1876699512827520/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761199.901672:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512827520/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761199.901687:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761199.901691:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761199.901696:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512827520:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761199.901746:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512827648:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761199.901764:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761199.901773:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761199/real 1789761199] req@ffff8dc005c95880 x1876699512827648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761199.901781:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512827648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761199.901787:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761199.901790:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761199.901792:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512827648:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761200.925099:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761201.949032:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761202.973084:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761203.997081:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761205.021025:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761205.021034:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761205.021038:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761205.021047:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1179 00000100:00080000:2.0:1789761205.021055:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761205.021062:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789761205.021081:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789761205.021096:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512827776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761205.021118:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761205.021121:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761205.021123:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761205.021127:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1179 00000100:00080000:2.0:1789761205.021129:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761205.021134:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761205.021144:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761205.021151:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512827776:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761205.021161:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512827904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761205.021174:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761205.021178:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761205.021180:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761205.021185:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1179 00000100:00080000:2.0:1789761205.021190:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761205.021195:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761205.021201:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761205.021208:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761205.021217:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512828032/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761205.021230:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761205.021234:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761205.021236:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761205.021242:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1179 00000100:00080000:2.0:1789761205.021246:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761205.021249:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761205/real 1789761205] req@ffff8dc00368df80 x1876699512827776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761205.021254:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761205.021263:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761205.021277:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512827776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761205.021280:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512828160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761205.021302:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761205.021309:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761205.021314:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512827776:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761205.021336:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512827904:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761205.021355:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761205.021365:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761205/real 1789761205] req@ffff8dc00368e300 x1876699512827904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761205.021373:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512827904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761205.021379:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761205.021381:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761205.021383:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512827904:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761205.021387:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512828032:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761205.021398:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761205.021405:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761205/real 1789761205] req@ffff8dc00368ed80 x1876699512828032/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761205.021412:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512828032/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761205.021417:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761205.021419:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761205.021421:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512828032:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761205.021424:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512828160:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761205.021434:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761205.021440:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761205/real 1789761205] req@ffff8dc00368f800 x1876699512828160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761205.021448:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512828160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761205.021453:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761205.021454:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761205.021456:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512828160:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761206.045101:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761207.069049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761208.093103:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761209.117095:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761210.141097:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761210.141154:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761210.141161:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761210.141166:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761210.141176:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1184 00000100:00080000:0.0:1789761210.141185:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761210.141192:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761210.141229:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149c00 x1876699512828288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761210.141259:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761210.141263:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761210.141265:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761210.141270:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1184 00000100:00080000:0.0:1789761210.141274:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761210.141278:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761210.141286:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761210.141298:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512828288:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761210.141299:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b480 x1876699512828416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761210.141319:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761210.141322:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761210.141323:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761210.141326:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1184 00000100:00080000:0.0:1789761210.141329:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761210.141331:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761210.141335:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761210.141345:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514bb80 x1876699512828544/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761210.141352:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789761210.141353:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1789761210.141358:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761210.141360:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761210.141364:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1184 00000100:00080000:0.0:1789761210.141367:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761210.141369:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761210.141373:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761210.141382:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148380 x1876699512828672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761210.141400:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761210/real 1789761210] req@ffff8dc005149c00 x1876699512828288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761210.141416:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149c00 x1876699512828288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761210.141423:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761210.141427:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761210.141430:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512828288:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761210.141449:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512828416:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761210.141463:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761210.141471:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761210/real 1789761210] req@ffff8dc00514b480 x1876699512828416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761210.141479:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b480 x1876699512828416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761210.141484:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761210.141486:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761210.141488:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512828416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761210.141492:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512828544:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761210.141502:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761210.141508:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761210/real 1789761210] req@ffff8dc00514bb80 x1876699512828544/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761210.141516:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514bb80 x1876699512828544/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761210.141521:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761210.141522:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761210.141524:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512828544:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761210.141528:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512828672:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761210.141538:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761210.141544:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761210/real 1789761210] req@ffff8dc005148380 x1876699512828672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761210.141551:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148380 x1876699512828672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761210.141556:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761210.141558:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761210.141560:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512828672:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761211.165069:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761212.189095:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761213.213114:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761214.237081:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761215.260983:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761215.260991:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761215.260994:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761215.261003:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1189 00000100:00080000:2.0:1789761215.261011:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761215.261016:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789761215.261043:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789761215.261049:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512828800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761215.261068:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761215.261071:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761215.261072:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761215.261075:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1189 00000100:00080000:2.0:1789761215.261077:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761215.261080:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761215.261087:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761215.261095:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512828928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761215.261103:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761215.261107:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1789761215.261108:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512828800:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789761215.261110:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761215.261116:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1189 00000100:00080000:2.0:1789761215.261118:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761215.261120:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761215.261124:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761215.261132:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512829056/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761215.261139:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761215.261142:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761215.261143:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761215.261146:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1189 00000100:00080000:2.0:1789761215.261157:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761215.261160:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761215.261163:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761215.261170:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512829184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761215.261205:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512828928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761215.261230:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512829056:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761215.261244:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512829184:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761216.285139:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761217.309153:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789761218.333289:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00010000:00010000:1.0:1789761218.333295:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789761218.333352:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789761218.333369:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789761218.333409:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789761218.333416:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789761218.333424:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:3.0:1789761218.333458:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761215/real 1789761218] req@ffff8dc00368ce00 x1876699512828928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761218.333488:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512828928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761218.333502:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761218.333508:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761218.333513:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512828928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761218.333523:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761215/real 1789761218] req@ffff8dc00368ca80 x1876699512829056/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761218.333536:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512829056/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761218.333546:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761218.333550:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761218.333554:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512829056:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761218.333562:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761215/real 1789761218] req@ffff8dc00368c000 x1876699512829184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761218.333574:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512829184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761218.333582:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761218.333586:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761218.333590:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512829184:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761218.333603:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761215/real 1789761218] req@ffff8dc00368ea00 x1876699512828800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761218.333617:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512828800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761218.333625:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761218.333628:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761218.333631:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512828800:192.168.203.156@tcp:250:kworker.0 00010000:00010000:2.0:1789761219.357089:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761220.381016:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761220.381031:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761220.381036:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761220.381049:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1194 00000100:00080000:2.0:1789761220.381062:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761220.381071:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761220.381123:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512829312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1789761220.381147:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761220.381161:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761220.381166:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761220.381169:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761220.381175:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1194 00000100:00080000:2.0:1789761220.381179:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761220.381184:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761220.381197:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761220.381212:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512829312:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761220.381214:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512829440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761220.381233:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761220.381236:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761220.381238:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761220.381241:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1194 00000100:00080000:2.0:1789761220.381244:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761220.381247:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761220.381251:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761220.381262:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512829568/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761220.381269:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761220.381272:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761220.381274:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761220.381277:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1194 00000100:00080000:2.0:1789761220.381279:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761220.381282:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761220.381286:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789761220.381295:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761220.381298:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512829696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761220.381360:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761220/real 1789761220] req@ffff8dc00368f480 x1876699512829312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761220.381385:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512829312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761220.381398:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761220.381403:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761220.381408:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512829312:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761220.381436:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512829440:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761220.381462:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761220.381476:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761220/real 1789761220] req@ffff8dc00368fb80 x1876699512829440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761220.381491:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512829440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761220.381500:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761220.381504:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761220.381507:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512829440:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761220.381514:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512829568:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761220.381538:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761220.381551:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761220/real 1789761220] req@ffff8dc00368d880 x1876699512829568/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761220.381565:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512829568/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761220.381581:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761220.381585:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761220.381588:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512829568:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761220.381595:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512829696:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761220.381618:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761220.381632:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761220/real 1789761220] req@ffff8dc00368f100 x1876699512829696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761220.381644:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512829696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761220.381652:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761220.381656:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761220.381659:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512829696:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761221.405167:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761222.429078:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761223.453057:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761224.477094:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761225.501025:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761225.501034:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761225.501038:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761225.501047:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1200 00000100:00080000:1.0:1789761225.501056:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761225.501062:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761225.501078:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761225.501096:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96d80 x1876699512829824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761225.501117:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761225.501120:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761225.501121:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761225.501124:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1200 00000100:00080000:1.0:1789761225.501126:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761225.501129:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761225.501137:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761225.501147:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512829952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761225.501151:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512829824:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761225.501162:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761225.501165:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761225.501166:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761225.501169:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1200 00000100:00080000:1.0:1789761225.501171:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761225.501173:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761225.501178:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761225.501186:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512830080/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761225.501194:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761225.501197:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761225.501198:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761225.501200:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1200 00000100:00080000:1.0:1789761225.501202:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761225.501205:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761225.501208:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761225.501214:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512830208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761225.501219:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761225.501255:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761225/real 1789761225] req@ffff8dc005c96d80 x1876699512829824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761225.501268:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96d80 x1876699512829824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761225.501276:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761225.501279:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761225.501281:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512829824:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761225.501299:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512829952:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761225.501315:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761225.501323:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761225/real 1789761225] req@ffff8dc005c95880 x1876699512829952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761225.501331:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512829952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761225.501337:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761225.501339:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761225.501341:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512829952:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761225.501345:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512830080:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761225.501355:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761225.501361:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761225/real 1789761225] req@ffff8dc005c96680 x1876699512830080/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761225.501369:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512830080/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761225.501377:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761225.501380:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761225.501384:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512830080:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761225.501389:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512830208:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761225.501414:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761225.501425:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761225/real 1789761225] req@ffff8dc005c97480 x1876699512830208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761225.501437:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512830208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761225.501444:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761225.501447:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761225.501450:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512830208:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761226.525068:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761227.549064:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761228.573070:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761229.597023:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761230.621000:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761230.621011:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761230.621015:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761230.621023:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1205 00000100:00080000:2.0:1789761230.621031:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761230.621037:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789761230.621040:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789761230.621071:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512830336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761230.621091:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761230.621094:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761230.621095:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761230.621098:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1205 00000100:00080000:2.0:1789761230.621100:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761230.621103:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761230.621111:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761230.621119:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512830336:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761230.621122:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512830464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761230.621133:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761230.621135:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761230.621136:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761230.621138:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1205 00000100:00080000:2.0:1789761230.621140:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761230.621142:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761230.621146:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761230.621154:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512830592/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761230.621166:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761230.621168:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761230.621170:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761230.621172:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1205 00000100:00080000:2.0:1789761230.621174:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789761230.621179:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789761230.621182:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761230.621186:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761230.621195:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512830720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761230.621217:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761230/real 1789761230] req@ffff8dc00368f100 x1876699512830336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761230.621231:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512830336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761230.621240:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761230.621242:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761230.621245:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512830336:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761230.621262:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512830464:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761230.621275:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761230.621283:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761230/real 1789761230] req@ffff8dc00368d880 x1876699512830464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761230.621290:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512830464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761230.621300:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761230.621302:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761230.621305:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512830464:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761230.621308:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512830592:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761230.621319:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761230.621325:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761230/real 1789761230] req@ffff8dc00368fb80 x1876699512830592/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761230.621332:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512830592/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761230.621338:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761230.621339:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761230.621342:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512830592:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761230.621346:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512830720:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761230.621355:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761230.621361:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761230/real 1789761230] req@ffff8dc00368f480 x1876699512830720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761230.621368:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512830720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761230.621373:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761230.621375:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761230.621378:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512830720:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761231.645073:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761232.669043:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761233.693015:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761234.717041:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761235.740995:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761235.741003:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761235.741008:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761235.741017:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1210 00010000:00010000:1.0:1789761235.741029:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761235.741030:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761235.741038:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761235.741072:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512830848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761235.741097:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761235.741101:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761235.741104:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761235.741109:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1210 00000100:00080000:3.0:1789761235.741113:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761235.741119:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512830848:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761235.741119:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761235.741130:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761235.741148:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512830976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761235.741157:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761235.741160:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761235.741162:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761235.741167:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1210 00000100:00080000:3.0:1789761235.741170:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761235.741174:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761235.741178:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761235.741187:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761235.741188:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512831104/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761235.741197:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761235.741200:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761235.741202:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761235.741206:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1210 00000100:00080000:3.0:1789761235.741208:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761235.741211:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761235.741215:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761235.741228:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512831232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761235.741230:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761235/real 1789761235] req@ffff8dc0107ad180 x1876699512830848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761235.741258:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512830848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761235.741269:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761235.741274:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761235.741278:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512830848:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761235.741306:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512830976:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761235.741328:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761235.741341:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761235/real 1789761235] req@ffff8dc0107af480 x1876699512830976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761235.741354:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512830976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761235.741363:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761235.741367:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761235.741370:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512830976:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761235.741377:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512831104:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761235.741396:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761235.741404:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761235/real 1789761235] req@ffff8dc0107adf80 x1876699512831104/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761235.741417:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512831104/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761235.741422:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761235.741424:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761235.741426:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512831104:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761235.741436:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512831232:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761235.741447:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761235.741453:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761235/real 1789761235] req@ffff8dc0107ac380 x1876699512831232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761235.741461:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512831232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761235.741466:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761235.741468:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761235.741470:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512831232:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761236.765020:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761237.789086:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761238.813014:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761239.837039:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761240.861050:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761240.861060:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761240.861064:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761240.861074:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1215 00000100:00080000:3.0:1789761240.861084:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761240.861091:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789761240.861099:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761240.861127:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512831360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761240.861162:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761240.861166:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761240.861169:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761240.861174:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1215 00000100:00080000:3.0:1789761240.861178:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761240.861182:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761240.861193:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512831360:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789761240.861193:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761240.861210:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512831488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761240.861222:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761240.861225:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761240.861227:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761240.861232:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1215 00000100:00080000:3.0:1789761240.861235:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761240.861239:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761240.861245:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761240.861252:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761240.861258:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512831616/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761240.861266:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761240.861269:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761240.861271:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761240.861274:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1215 00000100:00080000:3.0:1789761240.861283:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761240.861286:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761240.861290:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761240.861297:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761240/real 1789761240] req@ffff8dc0107af800 x1876699512831360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761240.861301:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512831744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761240.861337:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512831360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761240.861348:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761240.861352:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761240.861355:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512831360:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761240.861372:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512831488:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761240.861387:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761240.861396:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761240/real 1789761240] req@ffff8dc0107ae680 x1876699512831488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761240.861405:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512831488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761240.861418:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761240.861420:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761240.861422:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512831488:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761240.861426:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512831616:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761240.861439:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761240.861445:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761240/real 1789761240] req@ffff8dc0107aea00 x1876699512831616/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761240.861453:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512831616/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761240.861459:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761240.861461:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761240.861463:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512831616:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761240.861467:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512831744:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761240.861478:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761240.861484:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761240/real 1789761240] req@ffff8dc0107ad880 x1876699512831744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761240.861498:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512831744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761240.861503:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761240.861505:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761240.861507:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512831744:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761241.885066:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761242.909067:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761243.933064:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761244.957018:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761245.981011:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761245.981019:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761245.981022:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761245.981031:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1220 00010000:00010000:1.0:1789761245.981033:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761245.981039:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761245.981046:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761245.981078:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512831872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761245.981098:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761245.981102:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761245.981103:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761245.981107:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1220 00000100:00080000:2.0:1789761245.981111:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761245.981115:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761245.981124:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761245.981133:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512831872:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761245.981139:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512832000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761245.981151:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761245.981153:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761245.981154:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761245.981157:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1220 00000100:00080000:2.0:1789761245.981160:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761245.981164:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761245.981169:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761245.981179:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512832128/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761245.981188:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761245.981191:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1789761245.981195:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789761245.981196:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761245.981202:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1220 00000100:00080000:2.0:1789761245.981207:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761245.981211:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761245.981216:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761245.981226:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512832256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761245.981240:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761245/real 1789761245] req@ffff8dc00368ea00 x1876699512831872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761245.981262:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512831872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761245.981274:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761245.981280:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761245.981285:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512831872:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761245.981314:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512832000:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761245.981334:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761245.981344:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761245/real 1789761245] req@ffff8dc00368c000 x1876699512832000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761245.981353:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512832000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761245.981358:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761245.981360:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761245.981363:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512832000:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761245.981367:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512832128:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761245.981380:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761245.981387:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761245/real 1789761245] req@ffff8dc00368ca80 x1876699512832128/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761245.981394:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512832128/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761245.981399:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761245.981401:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761245.981403:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512832128:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761245.981406:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512832256:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761245.981416:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761245.981421:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761245/real 1789761245] req@ffff8dc00368ce00 x1876699512832256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761245.981428:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512832256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761245.981433:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761245.981435:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761245.981437:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512832256:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761247.005056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761248.029077:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761249.053113:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761250.077050:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761251.101048:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761251.101056:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761251.101059:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761251.101068:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1225 00000100:00080000:3.0:1789761251.101076:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761251.101083:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789761251.101115:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761251.101126:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512832384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761251.101157:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761251.101160:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761251.101161:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761251.101165:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1225 00000100:00080000:3.0:1789761251.101168:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761251.101173:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761251.101182:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761251.101194:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512832512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761251.101204:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761251.101206:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512832384:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761251.101208:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761251.101210:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761251.101223:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1225 00000100:00080000:3.0:1789761251.101225:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761251.101228:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761251.101232:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761251.101242:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512832640/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761251.101250:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761251.101253:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761251.101255:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761251.101258:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1225 00000100:00080000:3.0:1789761251.101261:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761251.101264:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761251.101269:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761251.101277:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512832768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761251.101304:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761251.101362:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761251/real 1789761251] req@ffff8dc0107ace00 x1876699512832384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761251.101386:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512832384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761251.101400:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761251.101404:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761251.101409:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512832384:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761251.101437:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512832512:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761251.101460:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761251.101473:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761251/real 1789761251] req@ffff8dc0107af100 x1876699512832512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761251.101487:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512832512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761251.101496:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761251.101499:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761251.101504:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512832512:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761251.101511:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512832640:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761251.101529:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761251.101539:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761251/real 1789761251] req@ffff8dc0107ad880 x1876699512832640/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761251.101551:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512832640/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761251.101571:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761251.101578:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761251.101581:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512832640:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761251.101588:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512832768:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761251.101607:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761251.101632:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761251/real 1789761251] req@ffff8dc0107aea00 x1876699512832768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761251.101646:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512832768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761251.101655:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761251.101659:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761251.101662:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512832768:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761252.125078:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761253.149057:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761254.173022:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761255.197059:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761256.221015:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761256.221025:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761256.221030:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761256.221041:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1230 00000100:00080000:3.0:1789761256.221056:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761256.221065:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789761256.221076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761256.221113:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512832896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761256.221144:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761256.221147:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761256.221149:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761256.221153:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1230 00000100:00080000:3.0:1789761256.221157:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761256.221161:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761256.221172:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761256.221184:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512832896:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761256.221192:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512833024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761256.221215:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761256.221219:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761256.221222:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761256.221227:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1230 00000100:00080000:3.0:1789761256.221233:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761256.221236:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761256.221244:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761256.221255:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761256.221259:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512833152/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761256.221272:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761256.221276:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761256.221278:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761256.221283:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1230 00000100:00080000:3.0:1789761256.221287:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761256.221290:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761256.221293:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761256.221308:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761256/real 1789761256] req@ffff8dc0107ae680 x1876699512832896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761256.221309:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512833280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761256.221358:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512832896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761256.221369:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761256.221372:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761256.221375:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512832896:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761256.221394:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512833024:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761256.221411:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761256.221419:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761256/real 1789761256] req@ffff8dc0107af800 x1876699512833024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761256.221427:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512833024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761256.221433:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761256.221435:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761256.221437:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512833024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761256.221442:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512833152:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761256.221457:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761256.221468:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761256/real 1789761256] req@ffff8dc0107ac380 x1876699512833152/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761256.221481:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512833152/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761256.221490:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761256.221494:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761256.221498:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512833152:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761256.221504:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512833280:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761256.221525:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761256.221539:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761256/real 1789761256] req@ffff8dc0107adf80 x1876699512833280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761256.221552:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512833280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761256.221561:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761256.221565:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761256.221569:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512833280:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761257.245103:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761258.269076:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761259.293071:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761260.317063:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761261.341040:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761261.341048:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761261.341052:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1789761261.341060:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761261.341062:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1235 00000100:00080000:1.0:1789761261.341077:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761261.341084:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761261.341115:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512833408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761261.341145:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761261.341147:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761261.341149:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761261.341151:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1235 00000100:00080000:1.0:1789761261.341154:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761261.341156:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761261.341163:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761261.341175:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512833536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761261.341187:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761261.341192:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789761261.341196:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512833408:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761261.341196:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761261.341208:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1235 00000100:00080000:1.0:1789761261.341213:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761261.341218:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761261.341226:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761261.341241:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512833664/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761261.341250:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761261.341252:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761261.341253:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761261.341257:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1235 00000100:00080000:1.0:1789761261.341270:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761261.341273:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761261.341276:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761261.341282:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761261.341287:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512833792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761261.341334:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761261/real 1789761261] req@ffff8dc005c94e00 x1876699512833408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761261.341351:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512833408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761261.341362:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761261.341365:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761261.341368:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512833408:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761261.341392:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512833536:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761261.341408:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761261.341417:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761261/real 1789761261] req@ffff8dc005c96300 x1876699512833536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761261.341425:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512833536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761261.341431:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761261.341433:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761261.341436:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512833536:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761261.341440:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512833664:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761261.341458:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761261.341469:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761261/real 1789761261] req@ffff8dc005c97b80 x1876699512833664/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761261.341483:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512833664/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761261.341499:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761261.341503:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761261.341507:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512833664:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761261.341513:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512833792:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761261.341531:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761261.341554:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761261/real 1789761261] req@ffff8dc005c94380 x1876699512833792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761261.341562:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512833792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761261.341568:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761261.341570:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761261.341572:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512833792:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761262.365060:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761263.389050:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761264.413107:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761265.437099:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761266.461035:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761266.461049:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761266.461055:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1789761266.461066:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761266.461071:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1241 00000100:00080000:1.0:1789761266.461085:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761266.461093:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761266.461144:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512833920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761266.461175:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761266.461180:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761266.461182:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761266.461188:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1241 00000100:00080000:1.0:1789761266.461195:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761266.461200:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761266.461212:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761266.461225:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512833920:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761266.461230:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512834048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761266.461250:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761266.461255:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761266.461258:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761266.461265:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1241 00000100:00080000:1.0:1789761266.461269:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761266.461274:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761266.461280:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761266.461294:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512834176/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761266.461314:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761266.461328:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761266.461334:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761266.461337:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761266.461344:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1241 00000100:00080000:1.0:1789761266.461350:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761266.461355:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761266.461362:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761266.461366:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761266/real 1789761266] req@ffff8dc005c94700 x1876699512833920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761266.461391:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512834304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761266.461398:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512833920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761266.461421:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761266.461428:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761266.461436:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512833920:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761266.461465:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512834048:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761266.461485:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761266.461495:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761266/real 1789761266] req@ffff8dc005c96a00 x1876699512834048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761266.461521:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512834048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761266.461527:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761266.461530:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761266.461532:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512834048:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761266.461537:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512834176:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761266.461553:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761266.461561:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761266/real 1789761266] req@ffff8dc005c95500 x1876699512834176/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761266.461568:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512834176/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761266.461574:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761266.461576:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761266.461578:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512834176:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761266.461581:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512834304:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761266.461592:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761266.461598:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761266/real 1789761266] req@ffff8dc005c95f80 x1876699512834304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761266.461606:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512834304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761266.461611:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761266.461613:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761266.461614:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512834304:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761267.485054:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761268.509076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761269.533018:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761270.557106:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761271.580993:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761271.581001:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1789761271.581002:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761271.581006:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761271.581017:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1246 00000100:00080000:2.0:1789761271.581027:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761271.581035:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761271.581071:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512834432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761271.581108:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761271.581113:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761271.581116:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761271.581121:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1246 00000100:00080000:2.0:1789761271.581127:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761271.581133:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761271.581143:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512834432:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789761271.581144:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761271.581168:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512834560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761271.581183:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761271.581187:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761271.581189:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761271.581192:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1246 00000100:00080000:2.0:1789761271.581196:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761271.581199:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789761271.581203:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:2.0:1789761271.581205:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761271.581218:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512834688/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761271.581227:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761271.581229:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761271.581231:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761271.581234:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1246 00000100:00080000:2.0:1789761271.581238:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761271.581239:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761271/real 1789761271] req@ffff8dc00368f800 x1876699512834432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761271.581243:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761271.581251:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761271.581261:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512834432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761271.581266:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512834816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761271.581278:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761271.581284:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761271.581289:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512834432:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761271.581307:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512834560:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761271.581322:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761271.581331:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761271/real 1789761271] req@ffff8dc00368ed80 x1876699512834560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761271.581339:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512834560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761271.581348:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761271.581351:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761271.581353:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512834560:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761271.581357:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512834688:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761271.581368:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761271.581374:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761271/real 1789761271] req@ffff8dc00368e300 x1876699512834688/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761271.581384:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512834688/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761271.581392:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761271.581396:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761271.581400:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512834688:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761271.581406:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512834816:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761271.581428:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761271.581442:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761271/real 1789761271] req@ffff8dc00368df80 x1876699512834816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761271.581457:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512834816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761271.581477:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761271.581480:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761271.581484:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512834816:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761272.605027:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761273.629028:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761274.653019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761275.677114:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761276.701002:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761276.701014:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761276.701018:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1789761276.701021:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761276.701029:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1251 00000100:00080000:1.0:1789761276.701039:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761276.701045:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761276.701076:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512834944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761276.701097:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761276.701101:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761276.701103:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761276.701107:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1251 00000100:00080000:1.0:1789761276.701111:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761276.701115:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761276.701124:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761276.701135:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512834944:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761276.701136:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512835072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761276.701147:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761276.701150:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761276.701151:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761276.701155:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1251 00000100:00080000:1.0:1789761276.701158:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761276.701161:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761276.701165:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761276.701174:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512835200/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761276.701183:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761276.701185:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761276.701187:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761276.701190:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1251 00000100:00080000:1.0:1789761276.701193:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761276.701194:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761276.701197:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761276.701201:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761276.701209:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512835328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761276.701241:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761276/real 1789761276] req@ffff8dc005c97800 x1876699512834944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761276.701259:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512834944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761276.701266:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761276.701269:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761276.701272:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512834944:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761276.701291:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512835072:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761276.701307:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761276.701316:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761276/real 1789761276] req@ffff8dc005c94a80 x1876699512835072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761276.701324:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512835072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761276.701329:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761276.701331:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761276.701333:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512835072:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761276.701337:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512835200:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761276.701347:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761276.701354:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761276/real 1789761276] req@ffff8dc005c95180 x1876699512835200/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761276.701364:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512835200/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761276.701373:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761276.701376:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761276.701379:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512835200:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761276.701385:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512835328:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761276.701404:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761276.701417:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761276/real 1789761276] req@ffff8dc005c95c00 x1876699512835328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761276.701444:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512835328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761276.701451:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761276.701453:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761276.701456:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512835328:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761277.725099:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761278.749026:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761279.773056:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761280.797056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761281.821004:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1789761281.821015:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761281.821015:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761281.821020:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761281.821032:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1256 00000100:00080000:1.0:1789761281.821042:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761281.821050:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761281.821086:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512835456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761281.821112:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761281.821116:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761281.821118:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761281.821123:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1256 00000100:00080000:1.0:1789761281.821127:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761281.821133:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761281.821143:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761281.821148:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512835456:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761281.821162:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512835584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761281.821176:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761281.821180:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761281.821183:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761281.821187:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1256 00000100:00080000:1.0:1789761281.821200:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761281.821205:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761281.821211:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761281.821226:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512835712/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761281.821247:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761281.821252:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512835584:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1789761281.821253:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761281.821258:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761281.821265:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1256 00000100:00080000:1.0:1789761281.821269:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761281.821275:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761281.821281:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761281.821289:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512835712:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761281.821297:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512835840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761281.821329:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512835840:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761282.845041:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761283.869075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761284.892994:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789761284.893180:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1789761284.893213:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789761284.893228:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761284.893268:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761284.893279:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761284.893289:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1789761284.893297:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761281/real 1789761284] req@ffff8dc005c95c00 x1876699512835584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761284.893325:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512835584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761284.893339:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761284.893344:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761284.893349:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512835584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761284.893357:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761281/real 1789761284] req@ffff8dc005c95180 x1876699512835712/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761284.893388:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512835712/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761284.893397:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761284.893401:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761284.893405:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512835712:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789761284.893412:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761281/real 1789761284] req@ffff8dc005c97100 x1876699512835456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761284.893426:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512835456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761284.893434:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761284.893438:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761284.893441:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512835456:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761284.893464:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761281/real 1789761284] req@ffff8dc005c94a80 x1876699512835840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761284.893479:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512835840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761284.893487:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761284.893491:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761284.893495:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512835840:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761285.917103:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761286.940993:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761286.941024:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761286.941030:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761286.941033:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761286.941040:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1261 00000100:00080000:2.0:1789761286.941047:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761286.941052:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761286.941080:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512835968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761286.941101:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761286.941104:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761286.941105:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761286.941108:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1261 00000100:00080000:2.0:1789761286.941111:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761286.941113:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761286.941121:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761286.941130:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512836096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761286.941146:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512835968:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789761286.941164:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761286.941168:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761286.941170:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761286.941175:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1261 00000100:00080000:2.0:1789761286.941179:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761286.941183:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761286.941189:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761286.941213:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512836224/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761286.941224:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789761286.941225:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761286.941232:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761286.941234:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761286.941240:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1261 00000100:00080000:2.0:1789761286.941245:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761286.941250:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761286.941256:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761286.941271:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512836352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761286.941280:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761286/real 1789761286] req@ffff8dc00368e680 x1876699512835968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761286.941302:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512835968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761286.941312:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761286.941317:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761286.941321:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512835968:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761286.941359:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512836096:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761286.941385:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761286.941398:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761286/real 1789761286] req@ffff8dc00368d180 x1876699512836096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761286.941410:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512836096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761286.941419:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761286.941422:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761286.941425:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512836096:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761286.941431:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512836224:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761286.941452:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761286.941463:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761286/real 1789761286] req@ffff8dc00368c700 x1876699512836224/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761286.941476:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512836224/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761286.941485:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761286.941488:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761286.941491:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512836224:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761286.941497:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512836352:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761286.941517:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761286.941529:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761286/real 1789761286] req@ffff8dc00368dc00 x1876699512836352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761286.941540:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512836352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761286.941548:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761286.941551:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761286.941554:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512836352:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761287.965048:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761288.989060:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761290.013093:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761291.037029:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761292.061022:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761292.061030:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761292.061033:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1789761292.061036:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761292.061042:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1266 00000100:00080000:1.0:1789761292.061052:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761292.061058:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761292.061088:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512836480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761292.061107:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761292.061110:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761292.061111:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761292.061114:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1266 00000100:00080000:1.0:1789761292.061116:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761292.061118:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761292.061127:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761292.061135:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512836608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761292.061142:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761292.061145:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761292.061148:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761292.061150:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512836480:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761292.061152:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1266 00000100:00080000:1.0:1789761292.061157:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761292.061163:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761292.061168:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761292.061183:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512836736/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761292.061194:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761292.061198:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761292.061200:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761292.061204:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1266 00000100:00080000:1.0:1789761292.061209:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761292.061214:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761292.061219:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761292.061221:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761292.061234:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512836864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761292.061276:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761292/real 1789761292] req@ffff8dc005c97800 x1876699512836480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761292.061295:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512836480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761292.061318:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761292.061323:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761292.061327:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512836480:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761292.061349:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512836608:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761292.061368:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761292.061377:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761292/real 1789761292] req@ffff8dc005c95f80 x1876699512836608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761292.061385:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512836608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761292.061391:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761292.061393:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761292.061395:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512836608:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761292.061399:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512836736:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761292.061411:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761292.061418:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761292/real 1789761292] req@ffff8dc005c95500 x1876699512836736/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761292.061425:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512836736/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761292.061430:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761292.061432:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761292.061434:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512836736:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761292.061437:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512836864:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761292.061447:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761292.061453:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761292/real 1789761292] req@ffff8dc005c96a00 x1876699512836864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761292.061460:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512836864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761292.061465:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761292.061467:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761292.061468:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512836864:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761293.085083:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761294.109028:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761295.133077:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761296.157109:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761297.181016:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761297.181024:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761297.181028:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761297.181038:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1271 00000100:00080000:2.0:1789761297.181049:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761297.181058:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789761297.181098:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789761297.181099:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512836992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761297.181128:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761297.181131:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761297.181133:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761297.181141:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1271 00000100:00080000:2.0:1789761297.181144:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761297.181148:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761297.181156:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761297.181165:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512836992:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761297.181168:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512837120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761297.181184:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761297.181186:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761297.181187:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761297.181190:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1271 00000100:00080000:2.0:1789761297.181192:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761297.181194:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761297.181197:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761297.181207:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512837248/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761297.181214:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761297.181217:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761297.181218:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761297.181221:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1271 00000100:00080000:2.0:1789761297.181224:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761297.181226:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761297.181229:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761297.181234:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761297.181241:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512837376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761297.181285:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761297/real 1789761297] req@ffff8dc00368dc00 x1876699512836992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761297.181300:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512836992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761297.181308:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761297.181311:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761297.181314:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512836992:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761297.181331:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512837120:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761297.181348:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761297.181357:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761297/real 1789761297] req@ffff8dc00368c700 x1876699512837120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761297.181365:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512837120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761297.181370:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761297.181372:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761297.181375:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512837120:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761297.181378:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512837248:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761297.181389:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761297.181395:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761297/real 1789761297] req@ffff8dc00368d180 x1876699512837248/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761297.181407:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512837248/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761297.181412:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761297.181414:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761297.181416:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512837248:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761297.181419:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512837376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761297.181429:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761297.181435:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761297/real 1789761297] req@ffff8dc00368e680 x1876699512837376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761297.181443:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512837376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761297.181447:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761297.181449:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761297.181451:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512837376:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761298.205040:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761299.229075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761300.253039:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761301.277016:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761302.301030:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1789761302.301032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761302.301042:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761302.301047:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761302.301065:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1276 00000100:00080000:1.0:1789761302.301073:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761302.301078:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761302.301109:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512837504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761302.301129:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761302.301132:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761302.301133:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761302.301136:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1276 00000100:00080000:1.0:1789761302.301138:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761302.301140:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761302.301148:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761302.301157:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512837632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761302.301164:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761302.301167:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761302.301169:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761302.301172:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512837504:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761302.301173:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1276 00000100:00080000:1.0:1789761302.301177:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761302.301183:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761302.301187:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761302.301194:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512837760/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761302.301201:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761302.301204:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761302.301205:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761302.301208:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1276 00000100:00080000:1.0:1789761302.301211:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761302.301215:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761302.301220:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761302.301248:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512837888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761302.301256:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761302.301307:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761302/real 1789761302] req@ffff8dc005c94700 x1876699512837504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761302.301328:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512837504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761302.301339:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761302.301344:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761302.301348:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512837504:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761302.301370:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512837632:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761302.301393:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761302.301406:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761302/real 1789761302] req@ffff8dc005c94380 x1876699512837632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761302.301420:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512837632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761302.301430:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761302.301434:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761302.301438:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512837632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761302.301445:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512837760:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761302.301466:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761302.301478:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761302/real 1789761302] req@ffff8dc005c97b80 x1876699512837760/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761302.301491:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512837760/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761302.301507:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761302.301511:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761302.301515:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512837760:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761302.301521:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512837888:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761302.301539:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761302.301550:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761302/real 1789761302] req@ffff8dc005c96300 x1876699512837888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761302.301563:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512837888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761302.301572:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761302.301575:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761302.301579:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512837888:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761303.325017:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761304.349032:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761305.373074:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761306.397020:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761307.421001:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761307.421010:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761307.421014:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761307.421023:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1282 00010000:00010000:3.0:1789761307.421026:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761307.421032:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761307.421039:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761307.421072:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512838016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761307.421092:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761307.421095:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761307.421097:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761307.421100:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1282 00000100:00080000:1.0:1789761307.421104:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761307.421107:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761307.421114:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761307.421127:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512838144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761307.421132:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512838016:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761307.421141:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761307.421145:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761307.421146:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761307.421150:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1282 00000100:00080000:1.0:1789761307.421154:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761307.421157:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761307.421161:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761307.421172:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512838272/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761307.421180:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761307.421183:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761307.421184:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761307.421188:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1282 00000100:00080000:1.0:1789761307.421202:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761307.421208:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789761307.421215:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1789761307.421216:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761307.421233:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512838400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761307.421257:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761307/real 1789761307] req@ffff8dc005c94e00 x1876699512838016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761307.421276:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512838016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761307.421288:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761307.421292:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761307.421297:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512838016:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761307.421320:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512838144:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761307.421344:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761307.421358:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761307/real 1789761307] req@ffff8dc005c97480 x1876699512838144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761307.421372:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512838144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761307.421382:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761307.421386:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761307.421397:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512838144:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761307.421404:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512838272:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761307.421426:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761307.421439:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761307/real 1789761307] req@ffff8dc005c96680 x1876699512838272/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761307.421454:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512838272/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761307.421464:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761307.421468:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761307.421472:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512838272:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761307.421479:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512838400:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761307.421502:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761307.421517:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761307/real 1789761307] req@ffff8dc005c95880 x1876699512838400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761307.421532:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512838400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761307.421541:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761307.421545:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761307.421550:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512838400:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761308.445046:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761309.469037:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761310.493100:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761311.517130:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761312.540977:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761312.540985:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761312.540988:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761312.540997:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1287 00000100:00080000:1.0:1789761312.541005:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761312.541011:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761312.541044:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96d80 x1876699512838528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761312.541063:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1789761312.541065:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761312.541066:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761312.541070:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761312.541073:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1287 00000100:00080000:1.0:1789761312.541076:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761312.541079:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761312.541087:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761312.541098:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512838528:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761312.541100:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512838656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761312.541116:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761312.541120:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761312.541122:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761312.541126:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1287 00000100:00080000:1.0:1789761312.541131:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761312.541136:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761312.541141:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761312.541167:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512838784/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761312.541171:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761312.541181:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761312.541184:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761312.541186:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761312.541190:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1287 00000100:00080000:1.0:1789761312.541193:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761312.541197:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761312.541202:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761312.541212:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512838912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761312.541216:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761312/real 1789761312] req@ffff8dc005c96d80 x1876699512838528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761312.541246:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96d80 x1876699512838528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761312.541256:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761312.541259:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761312.541262:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512838528:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761312.541281:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512838656:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761312.541297:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761312.541306:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761312/real 1789761312] req@ffff8dc005c95880 x1876699512838656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761312.541315:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512838656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761312.541325:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761312.541328:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761312.541330:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512838656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761312.541334:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512838784:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761312.541345:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761312.541351:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761312/real 1789761312] req@ffff8dc005c96680 x1876699512838784/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761312.541358:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512838784/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761312.541364:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761312.541365:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761312.541367:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512838784:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761312.541371:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512838912:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761312.541381:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761312.541387:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761312/real 1789761312] req@ffff8dc005c97480 x1876699512838912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761312.541394:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512838912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761312.541399:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761312.541401:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761312.541403:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512838912:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761313.565076:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761314.589011:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761315.613033:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761316.637053:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761317.660994:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761317.661002:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761317.661006:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1789761317.661010:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761317.661015:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1292 00000100:00080000:1.0:1789761317.661024:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761317.661030:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761317.661060:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512839040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761317.661078:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761317.661081:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761317.661082:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761317.661085:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1292 00000100:00080000:1.0:1789761317.661087:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761317.661089:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761317.661096:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761317.661107:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512839168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761317.661129:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512839040:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761317.661129:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761317.661138:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761317.661141:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761317.661145:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1292 00000100:00080000:1.0:1789761317.661148:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761317.661150:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761317.661154:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761317.661166:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512839296/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761317.661174:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761317.661177:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761317.661178:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761317.661180:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1292 00000100:00080000:1.0:1789761317.661183:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761317.661185:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761317.661188:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761317.661192:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761317.661197:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512839424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761317.661229:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761317/real 1789761317] req@ffff8dc005c94e00 x1876699512839040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761317.661243:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512839040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761317.661251:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761317.661254:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761317.661256:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512839040:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761317.661273:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512839168:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761317.661288:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761317.661297:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761317/real 1789761317] req@ffff8dc005c96300 x1876699512839168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761317.661304:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512839168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761317.661310:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761317.661312:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761317.661314:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512839168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761317.661318:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512839296:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761317.661330:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761317.661342:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761317/real 1789761317] req@ffff8dc005c97b80 x1876699512839296/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761317.661364:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512839296/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761317.661373:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761317.661376:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761317.661380:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512839296:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761317.661387:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512839424:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761317.661409:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761317.661422:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761317/real 1789761317] req@ffff8dc005c94380 x1876699512839424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761317.661435:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512839424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761317.661444:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761317.661447:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761317.661451:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512839424:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761318.685224:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761319.709041:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761320.733142:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761321.757052:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761322.781022:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761322.781034:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761322.781039:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1789761322.781050:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761322.781051:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1297 00000100:00080000:2.0:1789761322.781064:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761322.781071:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761322.781121:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512839552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761322.781149:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761322.781153:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761322.781155:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761322.781159:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1297 00000100:00080000:2.0:1789761322.781164:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761322.781169:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761322.781179:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761322.781181:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512839552:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761322.781194:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512839680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761322.781205:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761322.781209:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761322.781211:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761322.781215:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1297 00000100:00080000:2.0:1789761322.781219:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761322.781222:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761322.781228:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761322.781237:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761322.781242:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512839808/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761322.781259:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761322.781262:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761322.781264:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761322.781269:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1297 00000100:00080000:2.0:1789761322.781273:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761322.781274:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761322/real 1789761322] req@ffff8dc00368df80 x1876699512839552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761322.781282:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761322.781291:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761322.781297:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512839552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761322.781307:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512839936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761322.781309:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761322.781316:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761322.781324:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512839552:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761322.781346:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512839680:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761322.781367:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761322.781381:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761322/real 1789761322] req@ffff8dc00368e300 x1876699512839680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761322.781394:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512839680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761322.781404:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761322.781408:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761322.781411:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512839680:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761322.781417:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512839808:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761322.781434:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761322.781445:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761322/real 1789761322] req@ffff8dc00368ed80 x1876699512839808/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761322.781457:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512839808/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761322.781466:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761322.781469:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761322.781473:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512839808:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761322.781480:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512839936:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761322.781500:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761322.781509:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761322/real 1789761322] req@ffff8dc00368f800 x1876699512839936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761322.781517:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512839936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761322.781522:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761322.781524:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761322.781526:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512839936:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761323.805014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761324.829035:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761325.853051:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761326.877041:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761327.900993:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761327.901004:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761327.901009:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761327.901020:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1302 00010000:00010000:1.0:1789761327.901026:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761327.901033:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761327.901051:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761327.901082:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512840064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761327.901102:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761327.901105:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761327.901106:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761327.901109:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1302 00000100:00080000:2.0:1789761327.901112:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761327.901115:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761327.901122:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761327.901130:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512840192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761327.901137:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761327.901139:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761327.901141:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761327.901143:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1302 00000100:00080000:2.0:1789761327.901145:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761327.901147:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761327.901151:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761327.901153:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512840064:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761327.901161:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512840320/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761327.901169:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761327.901171:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761327.901172:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761327.901174:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1302 00000100:00080000:2.0:1789761327.901176:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761327.901179:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761327.901182:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761327.901190:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512840448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761327.901222:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761327.901273:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761327/real 1789761327] req@ffff8dc00368ce00 x1876699512840064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761327.901293:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512840064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761327.901304:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761327.901308:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761327.901312:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512840064:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761327.901321:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512840192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761327.901345:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761327.901358:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761327/real 1789761327] req@ffff8dc00368ca80 x1876699512840192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761327.901370:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512840192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761327.901378:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761327.901381:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761327.901384:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512840192:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761327.901408:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512840320:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761327.901431:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761327.901443:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761327/real 1789761327] req@ffff8dc00368c000 x1876699512840320/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761327.901457:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512840320/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761327.901466:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761327.901469:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761327.901473:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512840320:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761327.901479:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512840448:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761327.901500:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761327.901513:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761327/real 1789761327] req@ffff8dc00368ea00 x1876699512840448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761327.901527:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512840448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761327.901542:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761327.901545:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761327.901549:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512840448:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761328.925068:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761329.949053:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761330.973064:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761331.997046:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761333.020984:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761333.020993:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761333.020998:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761333.021023:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1307 00000100:00080000:2.0:1789761333.021039:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761333.021047:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789761333.021056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789761333.021085:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512840576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761333.021106:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761333.021110:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761333.021112:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761333.021116:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1307 00000100:00080000:2.0:1789761333.021120:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761333.021125:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761333.021134:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761333.021142:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512840576:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761333.021146:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512840704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761333.021156:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761333.021159:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761333.021161:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761333.021164:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1307 00000100:00080000:2.0:1789761333.021168:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761333.021171:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761333.021177:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761333.021186:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512840832/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761333.021195:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761333.021198:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1789761333.021199:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789761333.021200:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761333.021203:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1307 00000100:00080000:2.0:1789761333.021206:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761333.021209:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761333.021213:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761333.021222:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512840960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761333.021236:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761333/real 1789761333] req@ffff8dc00368f480 x1876699512840576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761333.021253:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512840576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761333.021265:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761333.021269:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761333.021271:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512840576:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761333.021290:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512840704:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761333.021308:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761333.021317:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761333/real 1789761333] req@ffff8dc00368fb80 x1876699512840704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761333.021325:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512840704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761333.021331:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761333.021333:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761333.021335:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512840704:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761333.021340:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512840832:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761333.021351:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761333.021357:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761333/real 1789761333] req@ffff8dc00368d880 x1876699512840832/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761333.021365:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512840832/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761333.021370:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761333.021372:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761333.021374:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512840832:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761333.021386:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512840960:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761333.021401:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761333.021412:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761333/real 1789761333] req@ffff8dc00368f100 x1876699512840960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761333.021425:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512840960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761333.021435:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761333.021438:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761333.021442:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512840960:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761334.045056:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761335.069070:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761336.093091:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761337.117092:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761338.141098:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761338.141109:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761338.141114:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761338.141125:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1312 00010000:00010000:0.0:1789761338.141131:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761338.141136:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761338.141146:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761338.141193:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512841088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761338.141218:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761338.141221:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761338.141222:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761338.141227:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1312 00000100:00080000:2.0:1789761338.141230:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761338.141234:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761338.141242:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761338.141255:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512841088:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761338.141266:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512841216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761338.141286:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761338.141291:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761338.141293:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761338.141299:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1312 00000100:00080000:2.0:1789761338.141304:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761338.141309:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761338.141317:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761338.141324:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761338.141343:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512841344/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761338.141359:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761338.141363:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761338/real 1789761338] req@ffff8dc00368f100 x1876699512841088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761338.141365:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761338.141370:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761338.141385:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1312 00000100:00080000:2.0:1789761338.141396:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761338.141407:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761338.141408:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512841088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789761338.141426:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761338.141429:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761338.141436:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761338.141441:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512841088:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761338.141444:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512841472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761338.141473:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512841216:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761338.141491:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761338.141499:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761338/real 1789761338] req@ffff8dc00368d880 x1876699512841216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761338.141508:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512841216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761338.141522:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761338.141524:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761338.141526:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512841216:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761338.141531:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512841344:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761338.141544:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761338.141551:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761338/real 1789761338] req@ffff8dc00368fb80 x1876699512841344/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761338.141559:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512841344/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761338.141569:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761338.141571:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761338.141573:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512841344:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761338.141577:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512841472:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761338.141588:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761338.141594:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761338/real 1789761338] req@ffff8dc00368f480 x1876699512841472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761338.141601:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512841472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761338.141606:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761338.141608:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761338.141610:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512841472:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761339.165092:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761340.189153:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761341.213054:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761342.237073:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761343.261019:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761343.261032:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761343.261037:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761343.261048:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1317 00000100:00080000:2.0:1789761343.261059:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761343.261067:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789761343.261085:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789761343.261104:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512841600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761343.261132:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761343.261136:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761343.261139:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761343.261144:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1317 00000100:00080000:2.0:1789761343.261149:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761343.261154:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761343.261161:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512841600:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789761343.261164:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761343.261181:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512841728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761343.261194:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761343.261198:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761343.261200:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761343.261205:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1317 00000100:00080000:2.0:1789761343.261210:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761343.261215:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789761343.261225:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761343.261233:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761343.261250:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512841856/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761343.261253:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761343/real 1789761343] req@ffff8dc00368ea00 x1876699512841600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761343.261278:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761343.261281:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1789761343.261285:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512841600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761343.261285:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761343.261291:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1317 00000100:00080000:2.0:1789761343.261298:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761343.261301:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761343.261304:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:0.0:1789761343.261308:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:2.0:1789761343.261311:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761343.261314:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512841600:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761343.261324:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512841984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761343.261336:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512841728:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761343.261353:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761343.261363:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761343/real 1789761343] req@ffff8dc00368c000 x1876699512841728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761343.261372:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512841728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761343.261379:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761343.261381:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761343.261384:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512841728:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761343.261387:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512841856:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761343.261406:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761343.261422:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761343/real 1789761343] req@ffff8dc00368ca80 x1876699512841856/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761343.261435:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512841856/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761343.261446:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761343.261451:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761343.261455:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512841856:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761343.261471:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512841984:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761343.261500:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761343.261513:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761343/real 1789761343] req@ffff8dc00368ce00 x1876699512841984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761343.261526:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512841984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761343.261535:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761343.261539:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761343.261545:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512841984:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761344.285032:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761345.309073:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761346.333038:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761347.357019:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761348.381064:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761348.381076:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761348.381081:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1789761348.381091:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761348.381093:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1322 00000100:00080000:2.0:1789761348.381110:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761348.381119:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761348.381162:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512842112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761348.381190:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761348.381193:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761348.381194:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761348.381198:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1322 00000100:00080000:2.0:1789761348.381202:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761348.381207:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761348.381219:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761348.381231:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512842112:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761348.381238:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512842240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761348.381269:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761348.381274:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761348.381276:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761348.381281:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1322 00000100:00080000:2.0:1789761348.381286:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761348.381292:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761348.381299:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761348.381316:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512842368/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761348.381330:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761348.381336:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761348.381340:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761348.381346:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1322 00000100:00080000:2.0:1789761348.381352:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761348.381356:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512842240:192.168.203.156@tcp:38:kworker.0 00000100:00080000:2.0:1789761348.381364:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761348.381380:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761348.381389:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512842368:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789761348.381400:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512842496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761348.381445:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512842496:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761349.405081:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761350.429076:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761351.453043:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789761351.453160:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1789761351.453203:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789761351.453220:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761351.453259:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761351.453268:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761351.453278:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1789761351.453315:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761348/real 1789761351] req@ffff8dc00368df80 x1876699512842496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761351.453343:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512842496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761351.453357:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761351.453362:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761351.453367:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512842496:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761351.453390:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761348/real 1789761351] req@ffff8dc00368ed80 x1876699512842240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761351.453405:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512842240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761351.453415:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761351.453420:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761351.453423:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512842240:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761351.453431:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761348/real 1789761351] req@ffff8dc00368e300 x1876699512842368/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761351.453442:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512842368/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761351.453450:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761351.453453:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761351.453456:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512842368:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761351.453462:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761348/real 1789761351] req@ffff8dc00368f800 x1876699512842112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761351.453474:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512842112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761351.453481:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761351.453484:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761351.453487:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512842112:192.168.203.156@tcp:250:kworker.0 00010000:00010000:1.0:1789761352.477035:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761353.501014:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761353.501024:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761353.501028:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761353.501039:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1328 00000100:00080000:1.0:1789761353.501049:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761353.501056:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789761353.501068:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761353.501089:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512842624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761353.501112:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761353.501115:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761353.501116:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761353.501120:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1328 00000100:00080000:1.0:1789761353.501124:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761353.501127:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761353.501135:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761353.501140:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512842624:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761353.501146:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512842752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761353.501155:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761353.501157:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761353.501159:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761353.501162:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1328 00000100:00080000:1.0:1789761353.501164:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761353.501166:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761353.501170:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761353.501178:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512842880/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761353.501187:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761353.501191:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761353.501193:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761353.501198:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761353.501203:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1328 00000100:00080000:1.0:1789761353.501208:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761353.501213:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761353.501219:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761353.501234:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761353/real 1789761353] req@ffff8dc005c94700 x1876699512842624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761353.501236:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512843008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761353.501266:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512842624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761353.501279:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761353.501284:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761353.501289:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512842624:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761353.501312:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512842752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761353.501335:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761353.501350:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761353/real 1789761353] req@ffff8dc005c96a00 x1876699512842752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761353.501361:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512842752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761353.501369:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761353.501373:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761353.501376:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512842752:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761353.501383:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512842880:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761353.501403:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761353.501415:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761353/real 1789761353] req@ffff8dc005c95500 x1876699512842880/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761353.501434:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512842880/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761353.501442:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761353.501446:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761353.501449:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512842880:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761353.501455:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512843008:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761353.501477:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761353.501489:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761353/real 1789761353] req@ffff8dc005c95f80 x1876699512843008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761353.501502:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512843008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761353.501510:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761353.501513:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761353.501516:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512843008:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761354.525028:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761355.549064:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761356.573011:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761357.597089:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761358.621039:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761358.621046:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761358.621051:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761358.621059:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1333 00000100:00080000:3.0:1789761358.621067:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761358.621073:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761358.621097:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761358.621107:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512843136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761358.621130:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761358.621133:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761358.621134:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761358.621137:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1333 00000100:00080000:3.0:1789761358.621139:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761358.621142:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761358.621149:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761358.621159:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512843264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761358.621166:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761358.621169:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761358.621171:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761358.621174:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1333 00000100:00080000:3.0:1789761358.621176:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761358.621178:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761358.621182:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761358.621193:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512843136:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761358.621193:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512843392/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761358.621248:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761358.621250:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761358.621252:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761358.621254:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1333 00000100:00080000:3.0:1789761358.621257:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761358.621259:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761358.621264:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761358.621273:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512843520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761358.621329:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761358.621409:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761358/real 1789761358] req@ffff8dc0107af480 x1876699512843136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761358.621432:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512843136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761358.621447:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761358.621452:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761358.621457:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512843136:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761358.621466:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512843264:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761358.621503:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761358.621516:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761358/real 1789761358] req@ffff8dc0107ad180 x1876699512843264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761358.621531:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512843264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761358.621541:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761358.621545:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761358.621549:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512843264:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761358.621579:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512843392:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761358.621601:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761358.621613:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761358/real 1789761358] req@ffff8dc0107aed80 x1876699512843392/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761358.621626:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512843392/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761358.621647:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761358.621651:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761358.621655:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512843392:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761358.621663:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512843520:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761358.621683:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761358.621695:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761358/real 1789761358] req@ffff8dc0107ad500 x1876699512843520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761358.621720:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512843520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761358.621729:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761358.621733:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761358.621736:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512843520:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761359.645054:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761360.669055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761361.693096:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761362.717015:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761363.741017:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761363.741031:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761363.741036:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761363.741047:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1338 00000100:00080000:3.0:1789761363.741058:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761363.741066:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761363.741071:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761363.741109:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512843648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761363.741134:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761363.741137:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761363.741139:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761363.741142:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1338 00000100:00080000:3.0:1789761363.741144:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761363.741147:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761363.741157:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761363.741167:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512843776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761363.741175:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761363.741179:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512843648:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761363.741180:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761363.741185:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761363.741193:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1338 00000100:00080000:3.0:1789761363.741197:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761363.741202:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761363.741208:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761363.741223:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512843904/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761363.741237:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761363.741241:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761363.741243:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761363.741248:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1338 00000100:00080000:3.0:1789761363.741254:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761363.741259:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761363.741264:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761363.741268:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761363.741281:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512844032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761363.741324:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761363/real 1789761363] req@ffff8dc0107ac000 x1876699512843648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761363.741346:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512843648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761363.741359:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761363.741363:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761363.741368:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512843648:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761363.741398:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512843776:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761363.741426:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761363.741439:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761363/real 1789761363] req@ffff8dc0107ac700 x1876699512843776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761363.741453:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512843776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761363.741461:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761363.741465:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761363.741469:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512843776:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761363.741476:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512843904:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761363.741499:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761363.741511:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761363/real 1789761363] req@ffff8dc0107ae300 x1876699512843904/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761363.741523:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512843904/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761363.741533:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761363.741536:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761363.741540:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512843904:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761363.741546:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512844032:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761363.741562:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761363.741571:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761363/real 1789761363] req@ffff8dc0107aca80 x1876699512844032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761363.741591:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512844032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761363.741599:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761363.741601:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761363.741603:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512844032:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761364.765045:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761365.789058:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761366.813018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761367.837040:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761368.860994:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761368.861004:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761368.861011:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761368.861015:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761368.861024:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1343 00000100:00080000:1.0:1789761368.861032:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761368.861037:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761368.861068:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512844160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761368.861088:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761368.861091:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761368.861092:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761368.861096:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1343 00000100:00080000:1.0:1789761368.861099:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761368.861102:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761368.861109:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761368.861119:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512844288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761368.861131:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761368.861137:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789761368.861138:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512844160:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761368.861139:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761368.861148:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1343 00000100:00080000:1.0:1789761368.861154:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761368.861158:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761368.861165:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761368.861179:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512844416/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761368.861187:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761368.861190:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761368.861191:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761368.861196:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1343 00000100:00080000:1.0:1789761368.861201:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761368.861208:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761368.861208:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761368.861217:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761368.861234:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512844544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761368.861253:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761368/real 1789761368] req@ffff8dc005c97800 x1876699512844160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761368.861273:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512844160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761368.861285:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761368.861289:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761368.861294:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512844160:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761368.861315:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512844288:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761368.861340:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761368.861354:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761368/real 1789761368] req@ffff8dc005c94a80 x1876699512844288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761368.861368:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512844288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761368.861379:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761368.861383:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761368.861387:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512844288:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761368.861394:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512844416:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761368.861416:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761368.861427:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761368/real 1789761368] req@ffff8dc005c97100 x1876699512844416/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761368.861440:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512844416/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761368.861449:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761368.861453:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761368.861456:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512844416:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761368.861463:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512844544:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761368.861486:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761368.861498:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761368/real 1789761368] req@ffff8dc005c95180 x1876699512844544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761368.861514:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512844544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761368.861523:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761368.861526:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761368.861531:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512844544:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761369.885124:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761370.909021:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761371.933183:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761372.957054:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761373.981016:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761373.981024:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761373.981028:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761373.981039:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1348 00000100:00080000:1.0:1789761373.981049:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761373.981056:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761373.981075:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761373.981092:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512844672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761373.981116:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761373.981121:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761373.981123:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761373.981125:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1348 00000100:00080000:1.0:1789761373.981128:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761373.981130:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761373.981142:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761373.981153:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512844800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761373.981163:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761373.981165:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512844672:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761373.981166:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761373.981168:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761373.981172:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1348 00000100:00080000:1.0:1789761373.981178:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761373.981182:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761373.981185:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761373.981192:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512844928/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761373.981199:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761373.981202:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761373.981203:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761373.981205:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1348 00000100:00080000:1.0:1789761373.981213:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761373.981215:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761373.981219:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761373.981226:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512845056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761373.981238:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761373.981278:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761373/real 1789761373] req@ffff8dc005c95c00 x1876699512844672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761373.981300:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512844672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761373.981311:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761373.981316:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761373.981321:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512844672:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761373.981343:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512844800:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761373.981362:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761373.981372:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761373/real 1789761373] req@ffff8dc005c95180 x1876699512844800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761373.981381:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512844800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761373.981387:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761373.981389:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761373.981391:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512844800:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761373.981395:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512844928:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761373.981408:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761373.981415:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761373/real 1789761373] req@ffff8dc005c97100 x1876699512844928/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761373.981424:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512844928/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761373.981433:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761373.981436:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761373.981439:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512844928:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761373.981445:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512845056:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761373.981468:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761373.981481:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761373/real 1789761373] req@ffff8dc005c94a80 x1876699512845056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761373.981494:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512845056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761373.981503:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761373.981507:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761373.981510:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512845056:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761375.005028:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761376.029061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761377.053084:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761378.077007:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761379.101041:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761379.101050:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761379.101055:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761379.101069:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1353 00000100:00080000:1.0:1789761379.101084:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761379.101091:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789761379.101127:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761379.101142:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512845184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761379.101169:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761379.101172:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761379.101176:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761379.101184:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1353 00000100:00080000:1.0:1789761379.101189:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761379.101194:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761379.101209:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761379.101234:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512845312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761379.101252:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761379.101261:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512845184:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761379.101262:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761379.101272:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761379.101284:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1353 00000100:00080000:1.0:1789761379.101292:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761379.101297:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761379.101304:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761379.101325:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512845440/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761379.101326:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761379.101354:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761379.101359:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1789761379.101361:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761379/real 1789761379] req@ffff8dc005c97800 x1876699512845184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761379.101363:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761379.101381:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1353 00000100:00080000:1.0:1789761379.101388:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761379.101391:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512845184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761379.101403:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1789761379.101408:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1789761379.101412:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789761379.101414:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761379.101427:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512845184:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761379.101440:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512845568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761379.101449:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512845312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761379.101471:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761379.101482:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761379/real 1789761379] req@ffff8dc005c95f80 x1876699512845312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761379.101492:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512845312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761379.101498:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761379.101501:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761379.101505:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512845312:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761379.101512:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512845440:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789761379.101534:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761379.101547:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761379/real 1789761379] req@ffff8dc005c95500 x1876699512845440/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761379.101560:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512845440/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761379.101570:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761379.101583:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761379.101588:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512845440:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789761379.101605:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512845568:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761379.101632:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761379.101647:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761379/real 1789761379] req@ffff8dc005c96a00 x1876699512845568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761379.101673:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512845568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761379.101684:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761379.101689:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761379.101693:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512845568:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761380.125023:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761381.149033:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761382.173019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761383.197079:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761384.221024:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761384.221035:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761384.221040:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1789761384.221044:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761384.221053:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1358 00000100:00080000:1.0:1789761384.221067:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761384.221077:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761384.221120:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512845696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761384.221143:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761384.221148:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761384.221150:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761384.221153:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1358 00000100:00080000:1.0:1789761384.221155:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761384.221157:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761384.221167:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761384.221179:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512845824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761384.221187:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761384.221190:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761384.221191:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761384.221194:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1358 00000100:00080000:1.0:1789761384.221196:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761384.221199:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761384.221204:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761384.221204:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512845696:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761384.221214:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512845952/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761384.221223:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761384.221226:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761384.221228:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761384.221232:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1358 00000100:00080000:1.0:1789761384.221235:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761384.221240:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761384.221246:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761384.221259:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512846080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761384.221315:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761384.221366:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761384/real 1789761384] req@ffff8dc005c94700 x1876699512845696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761384.221388:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512845696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761384.221397:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761384.221401:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761384.221404:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512845696:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761384.221411:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512845824:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761384.221431:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761384.221441:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761384/real 1789761384] req@ffff8dc005c94380 x1876699512845824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761384.221450:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512845824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761384.221458:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761384.221462:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761384.221464:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512845824:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761384.221486:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512845952:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789761384.221502:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761384.221509:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761384/real 1789761384] req@ffff8dc005c97b80 x1876699512845952/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761384.221517:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512845952/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761384.221523:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761384.221525:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761384.221535:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512845952:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789761384.221539:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512846080:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761384.221551:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761384.221558:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761384/real 1789761384] req@ffff8dc005c96300 x1876699512846080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761384.221566:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512846080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761384.221572:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761384.221574:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761384.221576:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512846080:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761385.245022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761386.269051:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761387.293046:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761388.316990:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761389.341046:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761389.341056:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761389.341060:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761389.341071:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1363 00000100:00080000:1.0:1789761389.341092:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761389.341102:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761389.341115:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761389.341138:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512846208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761389.341158:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761389.341161:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761389.341163:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761389.341166:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1363 00000100:00080000:1.0:1789761389.341169:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761389.341171:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761389.341178:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761389.341188:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512846336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761389.341196:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512846208:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761389.341198:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761389.341204:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761389.341206:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761389.341209:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1363 00000100:00080000:1.0:1789761389.341212:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761389.341214:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761389.341218:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761389.341226:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512846464/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761389.341233:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761389.341236:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761389.341237:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761389.341240:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1363 00000100:00080000:1.0:1789761389.341242:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761389.341244:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761389.341247:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761389.341255:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512846592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761389.341258:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761389.341301:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761389/real 1789761389] req@ffff8dc005c94e00 x1876699512846208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761389.341321:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512846208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761389.341329:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761389.341332:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761389.341334:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512846208:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761389.341352:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512846336:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761389.341367:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761389.341375:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761389/real 1789761389] req@ffff8dc005c97480 x1876699512846336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761389.341383:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512846336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761389.341389:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761389.341391:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761389.341392:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512846336:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761389.341396:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512846464:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789761389.341407:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761389.341413:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761389/real 1789761389] req@ffff8dc005c96680 x1876699512846464/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761389.341421:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512846464/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761389.341426:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761389.341428:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761389.341430:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512846464:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789761389.341433:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512846592:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761389.341443:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761389.341449:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761389/real 1789761389] req@ffff8dc005c95880 x1876699512846592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761389.341456:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512846592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761389.341461:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761389.341463:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761389.341464:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512846592:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761390.365019:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761391.389016:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761392.413112:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761393.437088:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761394.461031:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761394.461039:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761394.461043:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761394.461055:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1369 00010000:00010000:0.0:1789761394.461059:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761394.461066:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761394.461077:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761394.461115:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96d80 x1876699512846720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761394.461138:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761394.461143:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761394.461144:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761394.461147:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1369 00000100:00080000:1.0:1789761394.461150:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761394.461153:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761394.461160:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761394.461172:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512846848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761394.461179:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761394.461182:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761394.461183:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761394.461186:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1369 00000100:00080000:1.0:1789761394.461188:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761394.461192:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761394.461193:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512846720:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1789761394.461196:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761394.461205:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512846976/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761394.461212:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761394.461215:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761394.461216:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761394.461218:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1369 00000100:00080000:1.0:1789761394.461221:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761394.461224:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761394.461228:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761394.461235:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512847104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761394.461249:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761394.461284:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761394/real 1789761394] req@ffff8dc005c96d80 x1876699512846720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761394.461304:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96d80 x1876699512846720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761394.461312:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761394.461315:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761394.461318:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512846720:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761394.461323:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512846848:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761394.461338:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761394.461346:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761394/real 1789761394] req@ffff8dc005c95880 x1876699512846848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761394.461353:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512846848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761394.461359:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761394.461361:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761394.461363:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512846848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761394.461379:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512846976:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789761394.461390:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761394.461397:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761394/real 1789761394] req@ffff8dc005c96680 x1876699512846976/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761394.461404:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512846976/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761394.461409:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761394.461411:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761394.461413:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512846976:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789761394.461417:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512847104:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761394.461426:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761394.461432:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761394/real 1789761394] req@ffff8dc005c97480 x1876699512847104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761394.461439:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512847104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761394.461444:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761394.461446:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761394.461448:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512847104:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761395.485087:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761396.509080:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761397.533008:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761398.557105:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761399.580989:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761399.580996:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761399.581000:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761399.581009:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1374 00000100:00080000:1.0:1789761399.581016:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761399.581022:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761399.581036:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761399.581054:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512847232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761399.581074:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761399.581077:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761399.581078:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761399.581081:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1374 00000100:00080000:1.0:1789761399.581084:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761399.581087:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761399.581094:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761399.581104:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512847232:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761399.581105:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512847360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761399.581119:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761399.581121:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761399.581122:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761399.581124:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1374 00000100:00080000:1.0:1789761399.581127:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761399.581130:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761399.581133:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761399.581140:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512847488/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761399.581147:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761399.581149:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761399.581151:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761399.581154:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1374 00000100:00080000:1.0:1789761399.581156:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761399.581159:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761399.581162:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761399.581169:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512847616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761399.581177:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761399.581226:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761399/real 1789761399] req@ffff8dc005c94e00 x1876699512847232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761399.581248:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512847232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761399.581259:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761399.581264:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761399.581268:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512847232:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761399.581294:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512847360:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761399.581319:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761399.581333:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761399/real 1789761399] req@ffff8dc005c96300 x1876699512847360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761399.581347:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512847360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761399.581355:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761399.581359:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761399.581363:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512847360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761399.581369:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512847488:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789761399.581391:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761399.581404:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761399/real 1789761399] req@ffff8dc005c97b80 x1876699512847488/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761399.581417:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512847488/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761399.581425:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761399.581429:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761399.581432:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512847488:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789761399.581439:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512847616:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761399.581459:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761399.581471:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761399/real 1789761399] req@ffff8dc005c94380 x1876699512847616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761399.581484:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512847616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761399.581492:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761399.581495:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761399.581499:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512847616:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761400.605031:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761401.629062:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761402.653087:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761403.677011:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761404.700990:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761404.701000:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761404.701005:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761404.701016:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1379 00000100:00080000:1.0:1789761404.701026:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761404.701034:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761404.701072:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761404.701079:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512847744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761404.701098:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761404.701103:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761404.701105:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761404.701108:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1379 00000100:00080000:1.0:1789761404.701112:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761404.701116:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761404.701123:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761404.701125:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512847744:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761404.701139:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512847872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761404.701151:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761404.701154:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761404.701157:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761404.701162:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1379 00000100:00080000:1.0:1789761404.701166:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761404.701171:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761404.701178:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761404.701194:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512848000/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761404.701206:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761404.701210:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761404.701211:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761404.701219:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1379 00000800:00000100:2.0:1789761404.701219:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761404.701230:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761404.701235:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761404.701242:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761404.701254:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512848128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761404.701298:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761404/real 1789761404] req@ffff8dc005c94700 x1876699512847744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761404.701318:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512847744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761404.701329:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761404.701333:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761404.701336:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512847744:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761404.701361:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512847872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761404.701380:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761404.701390:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761404/real 1789761404] req@ffff8dc005c96a00 x1876699512847872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761404.701404:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512847872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761404.701411:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761404.701415:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761404.701419:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512847872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761404.701426:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512848000:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789761404.701449:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761404.701476:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761404/real 1789761404] req@ffff8dc005c95500 x1876699512848000/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761404.701492:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512848000/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761404.701502:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761404.701506:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761404.701510:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512848000:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789761404.701517:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512848128:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761404.701535:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761404.701544:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761404/real 1789761404] req@ffff8dc005c95f80 x1876699512848128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761404.701552:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512848128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761404.701563:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761404.701565:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761404.701567:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512848128:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761405.725061:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761406.749136:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761407.773053:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761408.797097:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761409.821012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761409.821023:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761409.821033:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761409.821038:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761409.821049:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1384 00000100:00080000:2.0:1789761409.821060:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761409.821067:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761409.821110:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512848256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761409.821143:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761409.821146:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761409.821147:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761409.821150:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1384 00000100:00080000:2.0:1789761409.821153:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761409.821155:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761409.821164:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761409.821174:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512848384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761409.821183:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761409.821188:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761409.821190:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761409.821195:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1384 00000100:00080000:2.0:1789761409.821199:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761409.821206:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761409.821207:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512848256:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789761409.821213:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761409.821228:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512848512/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761409.821241:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761409.821244:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761409.821246:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761409.821249:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1384 00000100:00080000:2.0:1789761409.821252:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761409.821255:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761409.821259:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761409.821268:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512848640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761409.821290:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761409.821351:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761409/real 1789761409] req@ffff8dc00368e680 x1876699512848256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761409.821370:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512848256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761409.821382:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761409.821387:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761409.821393:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512848256:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761409.821402:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512848384:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761409.821439:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761409.821455:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761409/real 1789761409] req@ffff8dc00368d180 x1876699512848384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761409.821470:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512848384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761409.821480:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761409.821483:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761409.821488:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512848384:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761409.821518:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512848512:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761409.821540:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761409.821550:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761409/real 1789761409] req@ffff8dc00368c700 x1876699512848512/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761409.821559:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512848512/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761409.821565:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761409.821567:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761409.821569:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512848512:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761409.821574:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512848640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761409.821585:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761409.821591:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761409/real 1789761409] req@ffff8dc00368dc00 x1876699512848640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761409.821599:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512848640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761409.821604:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761409.821606:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761409.821609:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512848640:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761410.845046:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761411.869117:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761412.893012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761413.917031:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761414.940986:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761414.940997:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761414.941001:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761414.941009:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1389 00000100:00080000:2.0:1789761414.941017:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761414.941023:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761414.941062:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512848768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761414.941095:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761414.941099:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761414.941101:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761414.941115:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1389 00000100:00080000:2.0:1789761414.941120:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761414.941125:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761414.941135:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761414.941149:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512848896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761414.941160:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761414.941161:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512848768:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789761414.941166:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761414.941170:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761414.941174:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1389 00000100:00080000:2.0:1789761414.941178:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761414.941182:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761414.941186:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00010000:00010000:0.0:1789761414.941195:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789761414.941196:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512849024/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761414.941207:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761414.941210:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761414.941212:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761414.941216:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1389 00000100:00080000:2.0:1789761414.941219:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761414.941222:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761414.941226:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761414.941235:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512849152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761414.941268:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512848896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761414.941308:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512849024:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761414.941343:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512849152:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761415.965101:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761416.989095:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761418.013230:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789761418.013334:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:1.0:1789761418.013406:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789761418.013425:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789761418.013484:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789761418.013494:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789761418.013502:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:0.0:1789761418.013744:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761414/real 1789761418] req@ffff8dc00368c700 x1876699512848896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761418.013789:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512848896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761418.013808:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761418.013816:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761418.013822:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512848896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761418.013835:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761414/real 1789761418] req@ffff8dc00368d180 x1876699512849024/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761418.013853:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512849024/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761418.013863:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761418.013867:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761418.013872:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512849024:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761418.013881:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761414/real 1789761418] req@ffff8dc00368e680 x1876699512849152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761418.013896:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512849152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761418.013950:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761418.013954:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761418.013958:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512849152:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761418.013968:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761414/real 1789761418] req@ffff8dc00368dc00 x1876699512848768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761418.013984:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512848768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761418.013992:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761418.013996:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761418.014000:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512848768:192.168.203.156@tcp:250:kworker.0 00010000:00010000:0.0:1789761419.037145:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761420.061055:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761420.061063:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761420.061066:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761420.061076:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1394 00000100:00080000:3.0:1789761420.061087:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761420.061095:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761420.061134:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699512849280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761420.061157:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761420.061161:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761420.061164:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761420.061169:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1394 00000100:00080000:3.0:1789761420.061173:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1789761420.061180:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761420.061180:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761420.061189:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761420.061206:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512849280:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761420.061211:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699512849408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761420.061229:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761420.061232:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761420.061233:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761420.061237:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1394 00000100:00080000:3.0:1789761420.061239:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761420.061241:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761420.061245:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761420.061256:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512849536/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761420.061263:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761420.061266:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761420.061268:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761420.061271:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1394 00000100:00080000:3.0:1789761420.061278:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761420.061281:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761420.061284:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761420.061294:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512849664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761420.061317:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761420.061395:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761420/real 1789761420] req@ffff8dc0107adc00 x1876699512849280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761420.061419:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699512849280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761420.061433:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761420.061438:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761420.061443:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512849280:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761420.061475:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512849408:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761420.061505:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761420.061519:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761420/real 1789761420] req@ffff8dc0107afb80 x1876699512849408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761420.061533:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699512849408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761420.061543:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761420.061547:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761420.061551:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512849408:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761420.061558:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512849536:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761420.061579:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761420.061590:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761420/real 1789761420] req@ffff8dc0107aca80 x1876699512849536/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761420.061601:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512849536/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761420.061610:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761420.061614:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761420.061617:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512849536:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761420.061624:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512849664:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761420.061644:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761420.061656:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761420/real 1789761420] req@ffff8dc0107ae300 x1876699512849664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761420.061670:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512849664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761420.061679:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761420.061683:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761420.061687:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512849664:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761421.085188:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761422.109062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761423.133040:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761424.157108:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761425.181013:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761425.181025:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761425.181029:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761425.181048:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1399 00000100:00080000:2.0:1789761425.181061:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761425.181069:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789761425.181089:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789761425.181128:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512849792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761425.181158:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761425.181163:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761425.181165:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761425.181170:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1399 00000100:00080000:2.0:1789761425.181174:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761425.181180:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761425.181193:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761425.181203:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512849792:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761425.181212:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512849920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761425.181225:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761425.181229:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761425.181231:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761425.181235:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1399 00000100:00080000:2.0:1789761425.181239:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761425.181244:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761425.181250:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761425.181263:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512850048/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761425.181274:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761425.181277:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761425.181279:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761425.181287:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1399 00000800:00000100:0.0:1789761425.181288:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789761425.181296:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761425.181302:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761425.181308:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761425.181337:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512850176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761425.181354:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761425/real 1789761425] req@ffff8dc00368f800 x1876699512849792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761425.181372:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512849792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761425.181381:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761425.181385:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761425.181389:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512849792:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761425.181411:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512849920:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761425.181428:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761425.181438:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761425/real 1789761425] req@ffff8dc00368e300 x1876699512849920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761425.181446:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512849920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761425.181452:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761425.181454:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761425.181457:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512849920:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761425.181460:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512850048:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761425.181472:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761425.181480:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761425/real 1789761425] req@ffff8dc00368ed80 x1876699512850048/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761425.181488:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512850048/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761425.181493:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761425.181496:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761425.181498:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512850048:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761425.181501:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512850176:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761425.181511:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761425.181517:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761425/real 1789761425] req@ffff8dc00368df80 x1876699512850176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761425.181524:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512850176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761425.181529:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761425.181531:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761425.181533:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512850176:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761426.205137:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761427.229051:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761428.253095:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761429.277093:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761430.301040:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761430.301054:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761430.301060:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761430.301076:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1404 00010000:00010000:2.0:1789761430.301086:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761430.301090:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761430.301102:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761430.301152:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512850304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761430.301185:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761430.301190:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761430.301191:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761430.301195:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1404 00000100:00080000:1.0:1789761430.301199:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761430.301204:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761430.301215:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761430.301229:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512850304:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761430.301232:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512850432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761430.301254:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761430.301260:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761430.301263:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761430.301281:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1404 00000100:00080000:1.0:1789761430.301287:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761430.301293:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761430.301298:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761430.301310:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512850560/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761430.301327:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789761430.301328:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761430.301335:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761430.301337:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761430.301342:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1404 00000100:00080000:1.0:1789761430.301345:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761430.301348:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761430.301352:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761430.301362:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512850688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761430.301394:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761430/real 1789761430] req@ffff8dc005c97800 x1876699512850304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761430.301408:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512850304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761430.301418:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761430.301421:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761430.301424:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512850304:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761430.301445:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512850432:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761430.301462:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761430.301471:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761430/real 1789761430] req@ffff8dc005c94a80 x1876699512850432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761430.301479:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512850432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761430.301485:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761430.301487:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761430.301489:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512850432:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761430.301493:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512850560:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761430.301503:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761430.301509:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761430/real 1789761430] req@ffff8dc005c97100 x1876699512850560/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761430.301516:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512850560/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761430.301521:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761430.301523:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761430.301525:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512850560:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761430.301529:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512850688:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761430.301540:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761430.301546:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761430/real 1789761430] req@ffff8dc005c95180 x1876699512850688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761430.301554:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512850688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761430.301558:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761430.301560:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761430.301562:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512850688:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761431.325084:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761432.349055:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761433.373087:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761434.397035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761435.420992:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761435.421001:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761435.421004:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761435.421013:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1410 00000100:00080000:1.0:1789761435.421022:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761435.421028:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761435.421058:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512850816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1789761435.421075:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761435.421081:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761435.421084:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761435.421086:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761435.421089:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1410 00000100:00080000:1.0:1789761435.421092:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761435.421096:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761435.421106:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761435.421117:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512850944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761435.421126:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512850816:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761435.421127:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761435.421134:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761435.421138:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761435.421142:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1410 00000100:00080000:1.0:1789761435.421144:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761435.421146:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761435.421149:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761435.421159:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512851072/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761435.421167:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761435.421169:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761435.421171:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761435.421173:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1410 00000100:00080000:1.0:1789761435.421184:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761435.421187:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761435.421190:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761435.421197:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512851200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761435.421211:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761435.421263:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761435/real 1789761435] req@ffff8dc005c95c00 x1876699512850816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761435.421276:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512850816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761435.421284:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761435.421286:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761435.421289:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512850816:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761435.421308:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512850944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761435.421324:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761435.421333:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761435/real 1789761435] req@ffff8dc005c95180 x1876699512850944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761435.421342:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512850944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761435.421347:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761435.421350:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761435.421352:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512850944:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761435.421356:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512851072:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761435.421366:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761435.421372:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761435/real 1789761435] req@ffff8dc005c97100 x1876699512851072/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761435.421380:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512851072/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761435.421385:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761435.421387:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761435.421389:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512851072:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761435.421392:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512851200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761435.421402:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761435.421408:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761435/real 1789761435] req@ffff8dc005c94a80 x1876699512851200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761435.421415:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512851200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761435.421424:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761435.421427:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761435.421429:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512851200:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761436.445094:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761437.469091:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761438.493083:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761439.517067:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761440.541000:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761440.541009:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761440.541013:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761440.541023:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1415 00000100:00080000:1.0:1789761440.541031:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761440.541038:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761440.541073:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512851328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1789761440.541096:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761440.541112:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761440.541118:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761440.541120:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761440.541124:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1415 00000100:00080000:1.0:1789761440.541128:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761440.541131:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761440.541140:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761440.541151:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512851456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761440.541160:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761440.541163:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761440.541164:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761440.541167:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1415 00000100:00080000:1.0:1789761440.541169:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761440.541173:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761440.541177:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761440.541183:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512851584/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761440.541193:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512851328:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761440.541210:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761440.541224:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761440.541229:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761440.541234:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1415 00000100:00080000:1.0:1789761440.541237:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761440.541239:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761440.541243:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761440.541253:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512851712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761440.541325:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761440.541385:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761440/real 1789761440] req@ffff8dc005c97800 x1876699512851328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761440.541406:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512851328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761440.541420:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761440.541425:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761440.541430:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512851328:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761440.541439:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512851456:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761440.541464:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761440.541479:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761440/real 1789761440] req@ffff8dc005c95f80 x1876699512851456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761440.541494:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512851456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761440.541516:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761440.541521:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761440.541525:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512851456:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761440.541558:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512851584:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761440.541582:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761440.541596:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761440/real 1789761440] req@ffff8dc005c95500 x1876699512851584/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761440.541609:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512851584/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761440.541618:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761440.541622:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761440.541626:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512851584:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761440.541634:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512851712:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761440.541656:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761440.541669:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761440/real 1789761440] req@ffff8dc005c96a00 x1876699512851712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761440.541683:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512851712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761440.541693:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761440.541696:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761440.541700:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512851712:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761441.565068:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761442.589056:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761443.613119:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761444.637044:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761445.660979:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761445.660990:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761445.660994:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761445.661006:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1420 00000100:00080000:1.0:1789761445.661016:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761445.661023:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761445.661051:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761445.661073:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512851840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761445.661097:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761445.661100:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761445.661102:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761445.661106:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1420 00000100:00080000:1.0:1789761445.661109:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761445.661112:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761445.661122:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761445.661137:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512851968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761445.661140:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512851840:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761445.661149:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761445.661152:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761445.661154:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761445.661170:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1420 00000100:00080000:1.0:1789761445.661173:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761445.661177:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761445.661182:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761445.661192:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512852096/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761445.661202:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761445.661206:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761445.661208:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761445.661212:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1420 00000100:00080000:1.0:1789761445.661218:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761445.661224:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761445.661229:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761445.661236:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761445.661245:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512852224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761445.661285:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761445/real 1789761445] req@ffff8dc005c94700 x1876699512851840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761445.661302:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512851840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761445.661311:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761445.661315:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761445.661318:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512851840:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761445.661345:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512851968:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761445.661370:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761445.661385:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761445/real 1789761445] req@ffff8dc005c94380 x1876699512851968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761445.661400:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512851968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761445.661410:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761445.661414:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761445.661418:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512851968:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761445.661424:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512852096:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761445.661449:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761445.661462:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761445/real 1789761445] req@ffff8dc005c97b80 x1876699512852096/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761445.661475:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512852096/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761445.661490:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761445.661494:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761445.661498:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512852096:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761445.661504:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512852224:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761445.661527:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761445.661540:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761445/real 1789761445] req@ffff8dc005c96300 x1876699512852224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761445.661553:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512852224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761445.661562:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761445.661566:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761445.661569:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512852224:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761446.685026:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761447.709074:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761448.733100:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761449.757103:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761450.781020:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761450.781028:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761450.781033:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761450.781042:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1425 00000100:00080000:2.0:1789761450.781050:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1789761450.781054:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761450.781057:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761450.781088:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512852352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761450.781110:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761450.781113:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761450.781115:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761450.781130:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1425 00000100:00080000:2.0:1789761450.781135:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761450.781139:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761450.781148:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761450.781157:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512852352:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761450.781163:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512852480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761450.781179:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761450.781182:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761450.781184:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761450.781187:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1425 00000100:00080000:2.0:1789761450.781191:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761450.781193:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761450.781197:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761450.781206:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512852608/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761450.781214:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761450.781217:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761450.781218:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761450.781220:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1425 00000100:00080000:2.0:1789761450.781223:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789761450.781228:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789761450.781229:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761450.781235:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761450.781243:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512852736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761450.781269:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761450/real 1789761450] req@ffff8dc00368ce00 x1876699512852352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761450.781347:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512852352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761450.781358:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761450.781363:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761450.781367:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512852352:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761450.781392:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512852480:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761450.781511:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761450.781527:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761450/real 1789761450] req@ffff8dc00368ca80 x1876699512852480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761450.781537:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512852480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761450.781545:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761450.781549:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761450.781596:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512852480:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761450.781607:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512852608:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761450.781631:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761450.781641:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761450/real 1789761450] req@ffff8dc00368c000 x1876699512852608/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761450.781649:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512852608/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761450.781655:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761450.781657:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761450.781659:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512852608:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761450.781663:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512852736:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761450.781675:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761450.781682:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761450/real 1789761450] req@ffff8dc00368ea00 x1876699512852736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761450.781689:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512852736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761450.781696:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761450.781699:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761450.781703:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512852736:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761451.805064:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761452.829013:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761453.853052:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761454.877072:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761455.900993:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761455.901002:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761455.901007:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761455.901016:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1430 00010000:00010000:1.0:1789761455.901022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761455.901024:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761455.901032:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761455.901063:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512852864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761455.901098:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761455.901101:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761455.901103:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761455.901107:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1430 00000100:00080000:2.0:1789761455.901111:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761455.901115:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761455.901123:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761455.901135:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512852864:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761455.901136:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512852992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761455.901150:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761455.901152:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761455.901154:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761455.901156:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1430 00000100:00080000:2.0:1789761455.901159:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761455.901161:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761455.901164:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761455.901176:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512853120/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761455.901194:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761455.901197:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761455.901200:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789761455.901205:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789761455.901216:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1430 00000100:00080000:2.0:1789761455.901221:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761455.901227:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761455.901232:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761455.901243:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512853248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761455.901252:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761455/real 1789761455] req@ffff8dc00368f480 x1876699512852864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761455.901272:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512852864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761455.901282:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761455.901286:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761455.901290:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512852864:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761455.901325:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512852992:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761455.901348:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761455.901362:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761455/real 1789761455] req@ffff8dc00368fb80 x1876699512852992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761455.901375:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512852992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761455.901383:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761455.901386:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761455.901390:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512852992:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761455.901396:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512853120:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761455.901417:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761455.901429:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761455/real 1789761455] req@ffff8dc00368d880 x1876699512853120/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761455.901440:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512853120/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761455.901455:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761455.901458:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761455.901462:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512853120:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761455.901468:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512853248:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761455.901487:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761455.901499:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761455/real 1789761455] req@ffff8dc00368f100 x1876699512853248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761455.901511:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512853248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761455.901520:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761455.901523:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761455.901526:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512853248:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761456.925066:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761457.949051:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761458.973030:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761459.997071:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761461.021087:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761461.021103:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761461.021109:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1789761461.021109:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761461.021123:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1435 00000100:00080000:1.0:1789761461.021136:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761461.021144:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761461.021189:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512853376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761461.021211:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761461.021214:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761461.021216:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761461.021219:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1435 00000100:00080000:1.0:1789761461.021222:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761461.021225:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761461.021234:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761461.021245:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512853504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761461.021254:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512853376:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761461.021269:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761461.021272:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761461.021274:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761461.021277:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1435 00000100:00080000:1.0:1789761461.021280:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761461.021282:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761461.021287:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761461.021295:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512853632/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761461.021303:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761461.021306:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761461.021307:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761461.021310:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1435 00000100:00080000:1.0:1789761461.021313:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761461.021316:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761461.021319:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761461.021325:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512853760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761461.021356:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761461.021408:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761461/real 1789761461] req@ffff8dc005c94e00 x1876699512853376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761461.021424:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512853376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761461.021433:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761461.021437:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761461.021440:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512853376:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761461.021463:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512853504:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761461.021481:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761461.021491:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761461/real 1789761461] req@ffff8dc005c97480 x1876699512853504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761461.021499:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512853504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761461.021505:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761461.021508:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761461.021510:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512853504:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761461.021514:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512853632:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761461.021534:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761461.021541:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761461/real 1789761461] req@ffff8dc005c96680 x1876699512853632/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761461.021549:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512853632/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761461.021555:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761461.021557:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761461.021559:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512853632:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761461.021562:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512853760:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761461.021575:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761461.021581:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761461/real 1789761461] req@ffff8dc005c95880 x1876699512853760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761461.021590:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512853760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761461.021595:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761461.021596:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761461.021598:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512853760:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761462.045017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761463.069084:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761464.093012:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761465.117062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761466.140995:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761466.141017:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761466.141021:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761466.141036:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1440 00010000:00010000:3.0:1789761466.141043:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761466.141050:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761466.141060:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761466.141104:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96d80 x1876699512853888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761466.141127:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761466.141130:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761466.141132:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761466.141135:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1440 00000100:00080000:1.0:1789761466.141137:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761466.141144:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761466.141153:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761466.141162:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512853888:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761466.141168:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512854016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761466.141177:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761466.141179:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761466.141180:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761466.141183:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1440 00000100:00080000:1.0:1789761466.141185:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761466.141188:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761466.141192:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761466.141202:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512854144/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761466.141210:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761466.141213:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761466.141214:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761466.141217:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1440 00000100:00080000:1.0:1789761466.141220:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761466.141223:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761466.141226:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761466.141234:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512854272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761466.141240:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761466.141295:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761466/real 1789761466] req@ffff8dc005c96d80 x1876699512853888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761466.141316:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96d80 x1876699512853888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761466.141336:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761466.141340:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761466.141345:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512853888:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761466.141373:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512854016:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761466.141397:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761466.141410:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761466/real 1789761466] req@ffff8dc005c95880 x1876699512854016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761466.141423:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512854016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761466.141432:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761466.141435:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761466.141439:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512854016:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761466.141446:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512854144:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761466.141468:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761466.141478:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761466/real 1789761466] req@ffff8dc005c96680 x1876699512854144/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761466.141487:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512854144/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761466.141496:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761466.141500:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761466.141503:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512854144:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761466.141510:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512854272:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761466.141531:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761466.141544:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761466/real 1789761466] req@ffff8dc005c97480 x1876699512854272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761466.141555:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512854272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761466.141566:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761466.141568:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761466.141570:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512854272:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761467.165068:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761468.189040:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761469.213065:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761470.237064:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761471.261054:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761471.261065:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761471.261070:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761471.261080:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1445 00010000:00010000:1.0:1789761471.261084:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761471.261092:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761471.261102:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761471.261141:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512854400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761471.261176:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761471.261181:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761471.261183:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761471.261188:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1445 00000100:00080000:2.0:1789761471.261193:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761471.261199:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761471.261206:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512854400:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789761471.261214:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761471.261234:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512854528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761471.261246:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761471.261250:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761471.261252:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761471.261256:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1445 00000100:00080000:2.0:1789761471.261261:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789761471.261261:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789761471.261270:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761471.261277:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761471.261291:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512854656/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761471.261303:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761471/real 1789761471] req@ffff8dc00368f100 x1876699512854400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761471.261304:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761471.261313:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761471.261317:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761471.261326:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1445 00000100:00080000:2.0:1789761471.261332:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761471.261339:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761471.261340:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512854400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789761471.261345:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1789761471.261360:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789761471.261365:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512854784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761471.261366:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761471.261372:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512854400:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761471.261398:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512854528:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761471.261419:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761471.261428:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761471/real 1789761471] req@ffff8dc00368d880 x1876699512854528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761471.261437:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512854528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761471.261452:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761471.261455:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761471.261458:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512854528:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761471.261463:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512854656:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761471.261478:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761471.261486:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761471/real 1789761471] req@ffff8dc00368fb80 x1876699512854656/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761471.261494:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512854656/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761471.261499:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761471.261501:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761471.261504:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512854656:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761471.261507:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512854784:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761471.261518:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761471.261524:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761471/real 1789761471] req@ffff8dc00368f480 x1876699512854784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761471.261531:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512854784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761471.261536:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761471.261538:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761471.261540:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512854784:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761472.285088:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761473.309079:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761474.333075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761475.357016:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761476.381030:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761476.381037:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761476.381041:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761476.381050:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1450 00000100:00080000:1.0:1789761476.381059:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1789761476.381059:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761476.381067:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761476.381116:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512854912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761476.381139:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761476.381142:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761476.381144:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761476.381146:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1450 00000100:00080000:1.0:1789761476.381149:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761476.381152:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761476.381160:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761476.381187:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512855040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761476.381196:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512854912:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761476.381197:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761476.381204:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761476.381209:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761476.381213:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1450 00000100:00080000:1.0:1789761476.381215:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761476.381217:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761476.381221:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761476.381229:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512855168/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761476.381236:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761476.381239:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761476.381240:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761476.381243:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1450 00000100:00080000:1.0:1789761476.381246:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761476.381248:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761476.381251:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789761476.381259:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761476.381260:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512855296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761476.381295:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761476/real 1789761476] req@ffff8dc005c94e00 x1876699512854912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761476.381308:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512854912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761476.381316:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761476.381319:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761476.381322:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512854912:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761476.381339:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512855040:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761476.381353:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761476.381361:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761476/real 1789761476] req@ffff8dc005c96300 x1876699512855040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761476.381370:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512855040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761476.381375:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761476.381378:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761476.381380:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512855040:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761476.381384:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512855168:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761476.381395:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761476.381400:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761476/real 1789761476] req@ffff8dc005c97b80 x1876699512855168/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761476.381413:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512855168/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761476.381418:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761476.381420:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761476.381422:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512855168:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761476.381426:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512855296:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761476.381436:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761476.381442:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761476/real 1789761476] req@ffff8dc005c94380 x1876699512855296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761476.381449:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512855296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761476.381454:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761476.381457:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761476.381459:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512855296:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761477.405053:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761478.429073:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761479.453046:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761480.477092:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761481.501038:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761481.501054:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761481.501060:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1789761481.501067:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761481.501070:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1456 00000100:00080000:1.0:1789761481.501083:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761481.501089:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761481.501121:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512855424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761481.501140:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761481.501143:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761481.501144:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761481.501147:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1456 00000100:00080000:1.0:1789761481.501150:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761481.501155:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761481.501163:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761481.501173:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512855552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761481.501179:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512855424:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761481.501197:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761481.501201:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761481.501204:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761481.501209:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1456 00000100:00080000:1.0:1789761481.501214:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761481.501218:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761481.501225:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761481.501242:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512855680/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761481.501255:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761481.501260:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761481.501262:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761481.501267:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1456 00000100:00080000:1.0:1789761481.501272:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761481.501275:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761481.501280:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761481.501294:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512855808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761481.501310:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512855552:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761481.501346:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512855680:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761481.501368:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512855808:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761482.525080:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761483.549026:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761484.573100:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789761484.573170:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1789761484.573212:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789761484.573229:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761484.573264:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761484.573272:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761484.573280:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:3.0:1789761484.573319:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761481/real 1789761484] req@ffff8dc005c96a00 x1876699512855552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761484.573348:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512855552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761484.573360:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761484.573364:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761484.573369:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512855552:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761484.573376:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761481/real 1789761484] req@ffff8dc005c95500 x1876699512855680/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761484.573389:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512855680/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761484.573397:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761484.573400:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761484.573403:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512855680:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761484.573410:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761481/real 1789761484] req@ffff8dc005c95f80 x1876699512855808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761484.573424:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512855808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761484.573433:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761484.573437:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761484.573441:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512855808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761484.573448:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761481/real 1789761484] req@ffff8dc005c94700 x1876699512855424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761484.573463:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512855424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761484.573472:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761484.573476:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761484.573480:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512855424:192.168.203.156@tcp:250:kworker.0 00010000:00010000:2.0:1789761485.597049:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761486.621038:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761486.621046:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761486.621050:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761486.621060:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1461 00010000:00010000:3.0:1789761486.621064:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761486.621071:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761486.621078:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761486.621114:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148000 x1876699512855936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761486.621137:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761486.621140:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761486.621141:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761486.621144:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1461 00000100:00080000:0.0:1789761486.621147:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761486.621153:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761486.621161:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761486.621171:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514ad80 x1876699512856064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761486.621183:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789761486.621184:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512855936:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789761486.621186:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761486.621190:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761486.621205:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1461 00000100:00080000:0.0:1789761486.621207:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761486.621210:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761486.621214:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761486.621228:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148700 x1876699512856192/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761486.621237:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761486.621240:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761486.621242:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761486.621245:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1461 00000100:00080000:0.0:1789761486.621250:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761486.621257:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761486.621266:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761486.621283:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514aa00 x1876699512856320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761486.621299:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761486.621367:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761486/real 1789761486] req@ffff8dc005148000 x1876699512855936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761486.621401:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148000 x1876699512855936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761486.621416:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761486.621423:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761486.621431:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512855936:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761486.621466:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512856064:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761486.621500:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761486.621516:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761486/real 1789761486] req@ffff8dc00514ad80 x1876699512856064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761486.621530:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514ad80 x1876699512856064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761486.621547:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761486.621551:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761486.621555:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512856064:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761486.621564:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512856192:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761486.621589:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761486.621603:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761486/real 1789761486] req@ffff8dc005148700 x1876699512856192/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761486.621617:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148700 x1876699512856192/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761486.621627:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761486.621631:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761486.621634:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512856192:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761486.621642:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512856320:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761486.621666:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761486.621679:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761486/real 1789761486] req@ffff8dc00514aa00 x1876699512856320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761486.621693:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514aa00 x1876699512856320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761486.621702:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761486.621706:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761486.621711:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512856320:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761487.645079:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761488.669034:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761489.693037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761490.717076:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761491.740983:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761491.740991:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761491.740994:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761491.741007:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1466 00000100:00080000:3.0:1789761491.741015:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761491.741021:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761491.741046:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761491.741057:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512856448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761491.741081:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761491.741083:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761491.741085:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761491.741088:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1466 00000100:00080000:3.0:1789761491.741090:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761491.741092:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761491.741099:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761491.741107:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512856576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761491.741115:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761491.741117:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761491.741119:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761491.741121:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1466 00000100:00080000:3.0:1789761491.741123:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761491.741127:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761491.741133:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512856448:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789761491.741133:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761491.741141:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512856704/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761491.741149:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761491.741151:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761491.741153:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761491.741155:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1466 00000100:00080000:3.0:1789761491.741157:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761491.741159:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761491.741163:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761491.741170:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512856832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761491.741194:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761491.741228:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761491/real 1789761491] req@ffff8dc0107ac700 x1876699512856448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761491.741241:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512856448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761491.741249:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761491.741252:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761491.741254:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512856448:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761491.741260:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512856576:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761491.741274:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761491.741286:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761491/real 1789761491] req@ffff8dc0107ac000 x1876699512856576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761491.741298:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512856576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761491.741308:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761491.741312:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761491.741316:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512856576:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761491.741338:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512856704:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761491.741362:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761491.741374:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761491/real 1789761491] req@ffff8dc0107ad500 x1876699512856704/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761491.741386:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512856704/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761491.741392:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761491.741395:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761491.741397:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512856704:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761491.741404:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512856832:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761491.741425:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761491.741437:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761491/real 1789761491] req@ffff8dc0107aed80 x1876699512856832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761491.741456:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512856832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761491.741461:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761491.741463:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761491.741466:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512856832:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761492.765038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761493.789163:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761494.813080:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761495.837136:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761496.861004:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761496.861019:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761496.861024:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761496.861035:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1471 00000100:00080000:3.0:1789761496.861047:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761496.861054:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761496.861060:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761496.861098:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512856960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761496.861121:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761496.861124:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761496.861127:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761496.861131:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1471 00000100:00080000:3.0:1789761496.861135:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761496.861138:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761496.861148:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761496.861156:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512856960:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761496.861160:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512857088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761496.861171:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761496.861173:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761496.861174:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761496.861177:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1471 00000100:00080000:3.0:1789761496.861180:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761496.861182:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761496.861186:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761496.861193:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512857216/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761496.861200:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761496.861203:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761496.861204:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761496.861207:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1471 00000100:00080000:3.0:1789761496.861211:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761496.861213:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761496.861216:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761496.861223:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512857344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761496.861244:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761496.861303:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761496/real 1789761496] req@ffff8dc0107ad180 x1876699512856960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761496.861320:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512856960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761496.861329:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761496.861333:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761496.861336:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512856960:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761496.861357:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512857088:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761496.861381:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761496.861391:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761496/real 1789761496] req@ffff8dc0107af480 x1876699512857088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761496.861400:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512857088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761496.861406:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761496.861408:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761496.861411:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512857088:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761496.861414:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512857216:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761496.861428:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761496.861436:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761496/real 1789761496] req@ffff8dc0107adf80 x1876699512857216/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761496.861443:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512857216/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761496.861449:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761496.861451:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761496.861452:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512857216:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761496.861456:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512857344:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761496.861467:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761496.861473:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761496/real 1789761496] req@ffff8dc0107ac380 x1876699512857344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761496.861481:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512857344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761496.861486:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761496.861488:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761496.861490:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512857344:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761497.885028:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761498.909045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761499.933051:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761500.957037:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761501.980991:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761501.980999:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761501.981003:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761501.981013:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1476 00000100:00080000:3.0:1789761501.981021:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761501.981027:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761501.981060:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512857472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1789761501.981062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761501.981083:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761501.981086:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761501.981087:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761501.981090:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1476 00000100:00080000:3.0:1789761501.981093:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761501.981096:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761501.981103:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761501.981112:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512857600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761501.981119:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512857472:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761501.981122:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761501.981128:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761501.981130:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761501.981133:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1476 00000100:00080000:3.0:1789761501.981135:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761501.981138:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761501.981142:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761501.981151:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512857728/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761501.981161:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761501.981165:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761501.981168:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761501.981172:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1476 00000100:00080000:3.0:1789761501.981176:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761501.981180:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761501.981186:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761501.981195:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761501.981200:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512857856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761501.981240:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761501/real 1789761501] req@ffff8dc0107af800 x1876699512857472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761501.981261:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512857472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761501.981273:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761501.981277:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761501.981282:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512857472:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761501.981304:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512857600:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761501.981326:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761501.981340:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761501/real 1789761501] req@ffff8dc0107ae680 x1876699512857600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761501.981353:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512857600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761501.981363:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761501.981366:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761501.981370:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512857600:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761501.981377:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512857728:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761501.981398:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761501.981410:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761501/real 1789761501] req@ffff8dc0107aea00 x1876699512857728/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761501.981422:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512857728/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761501.981430:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761501.981434:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761501.981437:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512857728:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761501.981450:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512857856:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761501.981470:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761501.981482:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761501/real 1789761501] req@ffff8dc0107ad880 x1876699512857856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761501.981494:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512857856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761501.981501:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761501.981504:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761501.981507:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512857856:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761503.005024:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761504.029054:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761505.053094:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761506.077043:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761507.100984:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761507.100992:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761507.100996:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761507.101004:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1481 00000100:00080000:1.0:1789761507.101014:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761507.101020:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761507.101028:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761507.101054:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512857984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761507.101075:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761507.101078:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761507.101080:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761507.101083:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1481 00000100:00080000:1.0:1789761507.101086:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761507.101089:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761507.101098:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761507.101106:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512857984:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761507.101110:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512858112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761507.101120:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761507.101122:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761507.101123:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761507.101126:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1481 00000100:00080000:1.0:1789761507.101129:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761507.101131:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761507.101134:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761507.101142:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512858240/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761507.101149:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761507.101152:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761507.101154:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761507.101156:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1481 00000100:00080000:1.0:1789761507.101158:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761507.101163:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789761507.101164:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1789761507.101167:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761507.101177:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512858368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761507.101207:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761507/real 1789761507] req@ffff8dc005c97800 x1876699512857984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761507.101222:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512857984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761507.101231:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761507.101234:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761507.101237:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512857984:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761507.101255:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512858112:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761507.101270:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761507.101279:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761507/real 1789761507] req@ffff8dc005c94a80 x1876699512858112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761507.101287:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512858112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761507.101293:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761507.101295:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761507.101297:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512858112:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761507.101301:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512858240:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761507.101312:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761507.101318:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761507/real 1789761507] req@ffff8dc005c97100 x1876699512858240/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761507.101325:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512858240/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761507.101330:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761507.101332:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761507.101334:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512858240:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761507.101338:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512858368:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761507.101348:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761507.101354:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761507/real 1789761507] req@ffff8dc005c95180 x1876699512858368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761507.101361:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512858368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761507.101371:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761507.101373:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761507.101375:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512858368:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761508.125103:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761509.149079:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761510.173095:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761511.197088:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761512.220987:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761512.220997:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761512.221002:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761512.221013:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1486 00000100:00080000:1.0:1789761512.221024:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761512.221031:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761512.221054:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761512.221070:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512858496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761512.221097:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761512.221101:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761512.221103:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761512.221108:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1486 00000100:00080000:1.0:1789761512.221113:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761512.221117:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761512.221127:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512858496:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1789761512.221128:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761512.221150:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512858624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761512.221162:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761512.221166:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761512.221168:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761512.221172:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1486 00000100:00080000:1.0:1789761512.221177:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761512.221181:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789761512.221187:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1789761512.221190:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761512.221205:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512858752/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761512.221217:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761512.221221:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1789761512.221225:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761512/real 1789761512] req@ffff8dc005c95c00 x1876699512858496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761512.221226:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761512.221236:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1486 00000100:00080000:1.0:1789761512.221243:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761512.221249:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512858496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761512.221250:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761512.221261:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761512.221263:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761512.221269:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761512.221273:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512858496:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761512.221278:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512858880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761512.221295:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512858624:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761512.221320:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761512.221330:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761512/real 1789761512] req@ffff8dc005c95180 x1876699512858624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761512.221338:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512858624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761512.221344:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761512.221346:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761512.221348:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512858624:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761512.221353:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512858752:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761512.221366:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761512.221376:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761512/real 1789761512] req@ffff8dc005c97100 x1876699512858752/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761512.221396:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512858752/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761512.221405:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761512.221408:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761512.221412:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512858752:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761512.221419:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512858880:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761512.221439:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761512.221447:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761512/real 1789761512] req@ffff8dc005c94a80 x1876699512858880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761512.221455:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512858880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761512.221461:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761512.221463:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761512.221465:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512858880:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761513.245084:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761514.269044:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761515.293050:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761516.317048:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761517.341000:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761517.341008:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761517.341012:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761517.341031:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1491 00000100:00080000:1.0:1789761517.341038:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761517.341044:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761517.341074:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512859008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761517.341101:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761517.341106:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761517.341109:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761517.341114:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1491 00000100:00080000:1.0:1789761517.341119:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761517.341123:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761517.341133:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761517.341148:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512859136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761517.341161:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761517.341163:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761517.341165:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761517.341168:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1491 00000100:00080000:1.0:1789761517.341171:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1789761517.341175:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761517.341179:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761517.341186:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761517.341196:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512859264/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761517.341202:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512859008:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761517.341207:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761517.341211:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761517.341212:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761517.341215:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1491 00000100:00080000:1.0:1789761517.341217:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761517.341220:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761517.341223:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761517.341232:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512859392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761517.341258:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761517.341289:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761517/real 1789761517] req@ffff8dc005c97800 x1876699512859008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761517.341329:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512859008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761517.341341:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761517.341345:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761517.341349:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512859008:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761517.341358:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512859136:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761517.341381:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761517.341391:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761517/real 1789761517] req@ffff8dc005c94700 x1876699512859136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761517.341400:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512859136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761517.341406:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761517.341408:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761517.341421:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512859136:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761517.341439:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512859264:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761517.341452:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761517.341459:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761517/real 1789761517] req@ffff8dc005c95f80 x1876699512859264/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761517.341467:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512859264/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761517.341472:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761517.341474:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761517.341476:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512859264:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761517.341480:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512859392:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761517.341490:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761517.341495:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761517/real 1789761517] req@ffff8dc005c95500 x1876699512859392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761517.341503:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512859392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761517.341508:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761517.341510:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761517.341512:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512859392:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761518.365023:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761519.389039:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761520.413014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761521.437070:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761522.461012:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761522.461025:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761522.461030:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761522.461042:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1497 00000100:00080000:3.0:1789761522.461053:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761522.461061:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761522.461067:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761522.461107:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512859520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761522.461129:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761522.461132:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761522.461135:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761522.461142:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1497 00000100:00080000:3.0:1789761522.461148:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761522.461154:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761522.461164:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761522.461173:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512859520:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761522.461188:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512859648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761522.461200:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761522.461203:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761522.461205:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761522.461209:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1497 00000100:00080000:3.0:1789761522.461213:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761522.461217:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761522.461225:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761522.461237:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512859776/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761522.461245:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761522.461248:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761522.461249:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761522.461256:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1497 00000800:00000100:2.0:1789761522.461258:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761522.461261:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761522.461267:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761522.461274:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761522.461288:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512859904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761522.461308:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761522/real 1789761522] req@ffff8dc0107af100 x1876699512859520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761522.461329:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512859520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761522.461338:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761522.461341:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761522.461344:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512859520:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761522.461367:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512859648:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761522.461399:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761522.461410:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761522/real 1789761522] req@ffff8dc0107ace00 x1876699512859648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761522.461418:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512859648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761522.461424:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761522.461426:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761522.461428:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512859648:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761522.461433:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512859776:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789761522.461452:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761522.461463:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761522/real 1789761522] req@ffff8dc0107ad880 x1876699512859776/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761522.461475:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512859776/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761522.461484:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761522.461487:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761522.461492:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512859776:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789761522.461508:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512859904:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761522.461532:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761522.461545:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761522/real 1789761522] req@ffff8dc0107aea00 x1876699512859904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761522.461554:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512859904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761522.461561:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761522.461563:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761522.461567:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512859904:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761523.485064:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761524.509048:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761525.533076:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761526.557031:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761527.580990:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761527.580999:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761527.581003:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761527.581020:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1502 00000100:00080000:3.0:1789761527.581030:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761527.581036:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761527.581054:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761527.581070:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512860032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761527.581088:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761527.581091:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761527.581093:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761527.581099:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1502 00000100:00080000:3.0:1789761527.581102:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761527.581105:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761527.581112:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512860032:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789761527.581112:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761527.581131:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512860160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761527.581139:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761527.581142:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761527.581143:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761527.581146:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1502 00000100:00080000:3.0:1789761527.581158:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761527.581160:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761527.581164:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:2.0:1789761527.581168:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761527.581177:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512860288/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761527.581186:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761527.581189:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761527.581190:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761527.581193:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1502 00000100:00080000:3.0:1789761527.581196:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761527.581199:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761527.581202:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761527.581207:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761527/real 1789761527] req@ffff8dc0107ae680 x1876699512860032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761527.581212:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512860416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761527.581239:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512860032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761527.581249:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761527.581253:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761527.581256:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512860032:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761527.581274:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512860160:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761527.581289:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761527.581297:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761527/real 1789761527] req@ffff8dc0107af800 x1876699512860160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761527.581305:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512860160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761527.581310:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761527.581312:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761527.581315:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512860160:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761527.581319:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512860288:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789761527.581330:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761527.581336:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761527/real 1789761527] req@ffff8dc0107ac380 x1876699512860288/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761527.581343:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512860288/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761527.581349:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761527.581350:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761527.581352:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512860288:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789761527.581356:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512860416:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761527.581366:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761527.581372:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761527/real 1789761527] req@ffff8dc0107adf80 x1876699512860416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761527.581379:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512860416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761527.581384:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761527.581386:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761527.581387:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512860416:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761528.605074:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761529.629073:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761530.653063:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761531.677062:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761532.701082:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761532.701175:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761532.701179:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761532.701183:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761532.701193:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1507 00000100:00080000:2.0:1789761532.701202:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761532.701207:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761532.701240:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512860544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761532.701265:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761532.701268:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761532.701269:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761532.701272:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1507 00000100:00080000:2.0:1789761532.701274:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761532.701277:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761532.701286:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761532.701301:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512860672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761532.701314:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761532.701317:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761532.701320:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761532.701327:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1507 00000100:00100000:3.0:1789761532.701330:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512860544:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789761532.701333:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761532.701344:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761532.701350:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761532.701362:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512860800/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761532.701372:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761532.701375:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761532.701377:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761532.701380:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1507 00000100:00080000:2.0:1789761532.701384:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761532.701389:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761532.701395:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761532.701403:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512860928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761532.701420:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761532.701468:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761532/real 1789761532] req@ffff8dc00368ea00 x1876699512860544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761532.701486:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512860544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761532.701495:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761532.701499:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761532.701503:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512860544:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761532.701541:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512860672:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761532.701562:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761532.701572:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761532/real 1789761532] req@ffff8dc00368c000 x1876699512860672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761532.701581:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512860672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761532.701587:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761532.701590:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761532.701592:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512860672:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761532.701596:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512860800:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761532.701610:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761532.701617:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761532/real 1789761532] req@ffff8dc00368ca80 x1876699512860800/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761532.701624:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512860800/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761532.701630:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761532.701632:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761532.701634:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512860800:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761532.701638:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512860928:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761532.701650:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761532.701657:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761532/real 1789761532] req@ffff8dc00368ce00 x1876699512860928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761532.701664:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512860928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761532.701669:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761532.701671:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761532.701674:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512860928:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761533.725070:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761534.749063:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761535.773066:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761536.797087:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761537.821011:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761537.821022:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761537.821027:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761537.821037:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1512 00000100:00080000:3.0:1789761537.821048:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761537.821054:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789761537.821070:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761537.821086:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512861056/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761537.821108:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761537.821112:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761537.821114:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761537.821119:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1512 00000100:00080000:3.0:1789761537.821123:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761537.821127:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761537.821137:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761537.821156:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512861056:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761537.821156:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512861184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761537.821174:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761537.821178:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761537.821179:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761537.821182:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1512 00000100:00080000:3.0:1789761537.821186:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761537.821189:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761537.821194:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761537.821204:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512861312/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761537.821211:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761537.821214:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761537.821215:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761537.821217:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1512 00000100:00080000:3.0:1789761537.821219:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761537.821222:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1789761537.821227:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789761537.821233:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761537.821243:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512861440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761537.821279:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761537/real 1789761537] req@ffff8dc0107af480 x1876699512861056/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761537.821297:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512861056/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761537.821309:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761537.821313:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761537.821317:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512861056:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761537.821341:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512861184:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761537.821364:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761537.821377:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761537/real 1789761537] req@ffff8dc0107ad180 x1876699512861184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761537.821390:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512861184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761537.821400:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761537.821403:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761537.821407:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512861184:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761537.821414:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512861312:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789761537.821435:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761537.821446:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761537/real 1789761537] req@ffff8dc0107aed80 x1876699512861312/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761537.821460:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512861312/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761537.821484:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761537.821488:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761537.821492:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512861312:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789761537.821499:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512861440:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761537.821519:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761537.821531:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761537/real 1789761537] req@ffff8dc0107ad500 x1876699512861440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761537.821545:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512861440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761537.821554:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761537.821557:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761537.821561:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512861440:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761538.845051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761539.869075:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761540.893054:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761541.917004:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761542.941004:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761542.941012:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761542.941015:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761542.941024:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1517 00010000:00010000:2.0:1789761542.941032:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761542.941032:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761542.941041:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761542.941072:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512861568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761542.941092:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761542.941095:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761542.941096:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761542.941099:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1517 00000100:00080000:3.0:1789761542.941101:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761542.941104:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761542.941112:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761542.941121:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512861696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761542.941126:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512861568:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761542.941143:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761542.941147:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761542.941149:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761542.941154:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1517 00000100:00080000:3.0:1789761542.941158:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761542.941162:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761542.941168:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761542.941181:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512861824/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761542.941192:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761542.941196:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761542.941200:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789761542.941201:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761542.941210:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1517 00000100:00080000:3.0:1789761542.941217:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761542.941222:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761542.941228:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761542.941242:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512861952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761542.941247:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761542/real 1789761542] req@ffff8dc0107ac000 x1876699512861568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761542.941275:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512861568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761542.941286:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761542.941291:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761542.941295:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512861568:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761542.941320:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512861696:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761542.941343:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761542.941355:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761542/real 1789761542] req@ffff8dc0107ac700 x1876699512861696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761542.941363:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512861696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761542.941377:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761542.941381:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761542.941385:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512861696:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761542.941392:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512861824:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789761542.941413:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761542.941426:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761542/real 1789761542] req@ffff8dc0107ae300 x1876699512861824/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761542.941453:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512861824/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761542.941462:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761542.941465:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761542.941469:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512861824:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789761542.941475:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512861952:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761542.941496:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761542.941508:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761542/real 1789761542] req@ffff8dc0107aca80 x1876699512861952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761542.941520:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512861952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761542.941528:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761542.941531:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761542.941535:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512861952:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761543.964997:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761544.989058:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761546.013050:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761547.037084:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761548.060991:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761548.061002:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761548.061007:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761548.061020:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1522 00000100:00080000:3.0:1789761548.061030:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761548.061037:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761548.061044:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761548.061077:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699512862080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761548.061103:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761548.061108:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761548.061110:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761548.061115:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1522 00000100:00080000:3.0:1789761548.061120:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761548.061128:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512862080:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761548.061129:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761548.061143:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761548.061161:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699512862208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761548.061182:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761548.061185:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761548.061188:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761548.061193:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1522 00000100:00080000:3.0:1789761548.061198:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761548.061203:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761548.061209:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761548.061216:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512862208:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761548.061226:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512862336/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761548.061241:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761548.061244:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512862336:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1789761548.061249:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761548.061253:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761548.061260:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1522 00000100:00080000:3.0:1789761548.061265:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761548.061270:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761548.061276:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761548.061291:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512862464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761548.061333:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512862464:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761549.085081:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761550.109100:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789761551.133360:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00010000:00010000:3.0:1789761551.133406:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789761551.133438:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789761551.133462:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789761551.133518:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789761551.133534:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789761551.133549:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:3.0:1789761551.133579:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761548/real 1789761551] req@ffff8dc0107ae300 x1876699512862464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761551.133618:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512862464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761551.133634:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761551.133641:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761551.133646:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512862464:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761551.133657:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761548/real 1789761551] req@ffff8dc0107aca80 x1876699512862336/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761551.133672:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512862336/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761551.133681:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761551.133685:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761551.133689:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512862336:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761551.133697:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761548/real 1789761551] req@ffff8dc0107adc00 x1876699512862208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761551.133710:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699512862208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761551.133717:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761551.133721:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761551.133725:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512862208:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761551.133732:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761548/real 1789761551] req@ffff8dc0107afb80 x1876699512862080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761551.133746:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699512862080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761551.133753:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761551.133757:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761551.133760:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512862080:192.168.203.156@tcp:250:kworker.0 00010000:00010000:0.0:1789761552.157156:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761553.181004:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761553.181012:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761553.181015:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761553.181024:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1527 00000100:00080000:1.0:1789761553.181032:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761553.181038:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789761553.181064:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761553.181070:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512862592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761553.181090:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761553.181094:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761553.181096:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761553.181102:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1527 00000100:00080000:1.0:1789761553.181106:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761553.181111:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761553.181119:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512862592:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1789761553.181121:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761553.181140:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512862720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761553.181153:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761553.181157:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761553.181160:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761553.181164:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1527 00000100:00080000:1.0:1789761553.181170:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761553.181174:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761553.181180:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761553.181194:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512862848/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761553.181203:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761553.181213:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761553.181216:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761553.181218:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761553.181223:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1527 00000100:00080000:1.0:1789761553.181226:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761553.181229:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761553.181234:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761553.181245:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512862976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761553.181254:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761553/real 1789761553] req@ffff8dc005c96a00 x1876699512862592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761553.181275:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512862592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761553.181285:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761553.181289:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761553.181293:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512862592:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761553.181317:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512862720:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761553.181341:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761553.181369:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761553/real 1789761553] req@ffff8dc005c94380 x1876699512862720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761553.181384:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512862720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761553.181392:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761553.181396:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761553.181400:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512862720:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761553.181407:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512862848:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761553.181432:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761553.181445:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761553/real 1789761553] req@ffff8dc005c97b80 x1876699512862848/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761553.181458:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512862848/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761553.181467:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761553.181471:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761553.181474:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512862848:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761553.181480:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512862976:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761553.181502:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761553.181514:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761553/real 1789761553] req@ffff8dc005c96300 x1876699512862976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761553.181527:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512862976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761553.181535:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761553.181538:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761553.181542:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512862976:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761554.205179:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761555.229042:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761556.253081:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761557.277056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761558.301022:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761558.301033:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761558.301037:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761558.301048:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1532 00000100:00080000:1.0:1789761558.301057:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761558.301065:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761558.301102:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512863104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761558.301125:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761558.301131:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761558.301134:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761558.301139:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1532 00000100:00080000:1.0:1789761558.301145:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761558.301150:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761558.301160:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761558.301175:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512863232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761558.301186:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761558.301189:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761558.301191:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761558.301195:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1532 00000100:00080000:1.0:1789761558.301199:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761558.301203:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761558.301208:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761558.301221:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512863360/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761558.301233:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761558.301237:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761558.301239:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761558.301244:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1532 00000100:00080000:1.0:1789761558.301247:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789761558.301251:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512863104:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761558.301255:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761558.301263:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761558.301289:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512863488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761558.301327:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00010000:00010000:1.0:1789761558.301344:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761558.301370:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761558/real 1789761558] req@ffff8dc005c94e00 x1876699512863104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761558.301447:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512863104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761558.301461:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761558.301464:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761558.301468:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512863104:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761558.301474:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512863232:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761558.301578:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761558.301590:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761558/real 1789761558] req@ffff8dc005c97480 x1876699512863232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761558.301640:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512863232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761558.301652:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761558.301656:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761558.301661:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512863232:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761558.301669:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512863360:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761558.301691:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761558.301701:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761558/real 1789761558] req@ffff8dc005c96680 x1876699512863360/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761558.301709:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512863360/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761558.301716:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761558.301718:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761558.301720:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512863360:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761558.301738:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512863488:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761558.301750:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761558.301757:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761558/real 1789761558] req@ffff8dc005c95880 x1876699512863488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761558.301765:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512863488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761558.301770:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761558.301772:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761558.301774:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512863488:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761559.325000:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761560.349040:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761561.373082:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761562.397048:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761563.420981:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:1.0:1789761563.421019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761563.421043:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761563.421048:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761563.421059:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1538 00000100:00080000:3.0:1789761563.421068:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761563.421075:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761563.421109:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699512863616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761563.421134:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761563.421138:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761563.421140:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761563.421146:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1538 00000100:00080000:3.0:1789761563.421150:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761563.421155:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761563.421163:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761563.421168:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512863616:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761563.421214:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699512863744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761563.421224:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761563.421228:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761563.421230:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761563.421233:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1538 00000100:00080000:3.0:1789761563.421236:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761563.421240:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789761563.421244:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789761563.421245:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761563.421254:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512863872/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761563.421264:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761563.421267:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761563.421268:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761563.421272:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1538 00000100:00080000:3.0:1789761563.421291:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761563.421295:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761563/real 1789761563] req@ffff8dc0107afb80 x1876699512863616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761563.421296:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761563.421301:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761563.421312:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512864000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761563.421313:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699512863616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761563.421341:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761563.421346:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761563.421350:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512863616:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761563.421373:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512863744:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761563.421394:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761563.421407:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761563/real 1789761563] req@ffff8dc0107adc00 x1876699512863744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761563.421421:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699512863744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761563.421430:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761563.421434:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761563.421439:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512863744:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761563.421446:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512863872:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761563.421468:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761563.421477:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761563/real 1789761563] req@ffff8dc0107aca80 x1876699512863872/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761563.421486:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512863872/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761563.421508:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761563.421510:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761563.421512:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512863872:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761563.421516:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512864000:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761563.421529:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761563.421536:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761563/real 1789761563] req@ffff8dc0107ae300 x1876699512864000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761563.421543:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512864000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761563.421548:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761563.421550:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761563.421552:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512864000:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761564.445054:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761565.469029:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761566.493065:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761567.517063:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761568.540985:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761568.540993:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761568.540997:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761568.541005:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1543 00000100:00080000:3.0:1789761568.541013:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761568.541019:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789761568.541046:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761568.541054:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512864128/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761568.541070:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761568.541073:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761568.541075:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761568.541078:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1543 00000100:00080000:3.0:1789761568.541081:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761568.541084:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761568.541091:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761568.541099:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512864128:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761568.541102:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512864256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761568.541114:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761568.541117:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761568.541118:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761568.541121:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1543 00000100:00080000:3.0:1789761568.541123:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761568.541125:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761568.541128:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761568.541135:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512864384/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761568.541142:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761568.541144:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761568.541145:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761568.541148:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1543 00000100:00080000:3.0:1789761568.541158:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761568.541161:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761568.541164:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761568.541173:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761568.541174:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512864512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761568.541220:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761568/real 1789761568] req@ffff8dc0107ac700 x1876699512864128/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761568.541257:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512864128/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761568.541269:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761568.541274:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761568.541278:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512864128:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761568.541302:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512864256:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761568.541337:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761568.541354:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761568/real 1789761568] req@ffff8dc0107ac000 x1876699512864256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761568.541369:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512864256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761568.541378:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761568.541382:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761568.541386:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512864256:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761568.541392:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512864384:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761568.541415:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761568.541429:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761568/real 1789761568] req@ffff8dc0107ad500 x1876699512864384/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761568.541451:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512864384/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761568.541459:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761568.541463:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761568.541466:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512864384:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761568.541473:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512864512:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761568.541496:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761568.541510:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761568/real 1789761568] req@ffff8dc0107aed80 x1876699512864512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761568.541521:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512864512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761568.541527:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761568.541529:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761568.541531:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512864512:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761569.565005:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761570.589054:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761571.613077:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761572.636993:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761573.661032:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761573.661045:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761573.661051:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761573.661065:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1548 00010000:00010000:2.0:1789761573.661067:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761573.661080:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761573.661089:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761573.661143:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96d80 x1876699512864640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761573.661173:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761573.661177:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761573.661179:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761573.661184:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1548 00000100:00080000:1.0:1789761573.661189:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761573.661195:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761573.661219:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512864640:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1789761573.661220:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761573.661242:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512864768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761573.661262:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761573.661266:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761573.661268:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761573.661274:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1548 00000100:00080000:1.0:1789761573.661290:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761573.661295:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761573.661301:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761573.661308:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761573.661316:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512864896/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761573.661329:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761573.661334:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761573.661336:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761573.661344:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1548 00000100:00080000:1.0:1789761573.661349:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761573.661354:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761573.661358:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761573/real 1789761573] req@ffff8dc005c96d80 x1876699512864640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789761573.661362:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761573.661380:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512865024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761573.661383:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96d80 x1876699512864640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761573.661413:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761573.661418:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761573.661423:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512864640:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761573.661446:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512864768:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761573.661463:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761573.661472:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761573/real 1789761573] req@ffff8dc005c95880 x1876699512864768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761573.661481:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512864768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761573.661487:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761573.661489:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761573.661491:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512864768:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761573.661495:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512864896:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761573.661506:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761573.661513:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761573/real 1789761573] req@ffff8dc005c96680 x1876699512864896/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761573.661520:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512864896/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761573.661526:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761573.661527:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761573.661529:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512864896:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761573.661697:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512865024:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761573.661765:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761573.661791:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761573/real 1789761573] req@ffff8dc005c97480 x1876699512865024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761573.661806:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512865024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761573.661818:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761573.661823:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761573.661828:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512865024:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761574.685039:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761575.709012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761576.733036:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761577.757046:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761578.780985:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761578.780993:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761578.780996:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789761578.780999:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761578.781005:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1553 00000100:00080000:1.0:1789761578.781014:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761578.781019:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761578.781049:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512865152/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761578.781068:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761578.781071:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761578.781072:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761578.781075:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1553 00000100:00080000:1.0:1789761578.781078:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761578.781081:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761578.781088:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761578.781096:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512865152:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761578.781101:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512865280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761578.781110:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761578.781112:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761578.781113:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761578.781116:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1553 00000100:00080000:1.0:1789761578.781118:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761578.781121:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761578.781126:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761578.781140:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512865408/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761578.781152:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789761578.781153:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761578.781158:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761578.781174:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761578.781181:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1553 00000100:00080000:1.0:1789761578.781185:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761578.781189:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761578.781193:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761578.781213:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512865536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761578.781214:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761578/real 1789761578] req@ffff8dc005c94e00 x1876699512865152/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761578.781252:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512865152/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761578.781264:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761578.781269:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761578.781273:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512865152:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761578.781296:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512865280:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761578.781321:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761578.781331:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761578/real 1789761578] req@ffff8dc005c96300 x1876699512865280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761578.781340:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512865280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761578.781346:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761578.781354:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761578.781356:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512865280:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761578.781361:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512865408:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761578.781373:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761578.781380:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761578/real 1789761578] req@ffff8dc005c97b80 x1876699512865408/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761578.781388:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512865408/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761578.781393:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761578.781395:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761578.781396:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512865408:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761578.781400:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512865536:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761578.781410:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761578.781416:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761578/real 1789761578] req@ffff8dc005c94380 x1876699512865536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761578.781428:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512865536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761578.781433:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761578.781435:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761578.781437:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512865536:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761579.805125:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761580.829027:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761581.853047:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761582.877017:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761583.901015:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761583.901025:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761583.901030:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1789761583.901036:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761583.901041:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1558 00000100:00080000:2.0:1789761583.901052:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761583.901058:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761583.901097:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512865664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761583.901116:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761583.901134:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761583.901136:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761583.901140:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1558 00000100:00080000:2.0:1789761583.901143:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761583.901147:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761583.901154:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761583.901164:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512865664:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761583.901166:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512865792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761583.901180:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761583.901182:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761583.901183:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761583.901186:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1558 00000100:00080000:2.0:1789761583.901188:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761583.901190:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761583.901193:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761583.901202:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512865920/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761583.901210:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761583.901212:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761583.901213:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761583.901215:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1558 00000100:00080000:2.0:1789761583.901218:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761583.901220:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761583.901224:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761583.901231:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512866048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761583.901236:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761583.901279:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761583/real 1789761583] req@ffff8dc00368df80 x1876699512865664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761583.901298:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512865664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761583.901309:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761583.901313:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761583.901317:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512865664:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761583.901339:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512865792:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761583.901361:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761583.901374:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761583/real 1789761583] req@ffff8dc00368ed80 x1876699512865792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761583.901387:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512865792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761583.901411:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761583.901415:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761583.901418:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512865792:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761583.901425:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512865920:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761583.901446:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761583.901457:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761583/real 1789761583] req@ffff8dc00368e300 x1876699512865920/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761583.901469:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512865920/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761583.901478:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761583.901481:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761583.901485:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512865920:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761583.901491:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512866048:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761583.901512:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761583.901535:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761583/real 1789761583] req@ffff8dc00368f800 x1876699512866048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761583.901548:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512866048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761583.901557:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761583.901560:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761583.901564:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512866048:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761584.925005:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761585.949034:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761586.973058:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761587.997024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761589.021041:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761589.021052:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761589.021057:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761589.021069:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1563 00010000:00010000:3.0:1789761589.021074:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761589.021081:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761589.021102:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761589.021139:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512866176/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761589.021166:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761589.021170:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761589.021172:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761589.021176:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1563 00000100:00080000:2.0:1789761589.021182:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761589.021186:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761589.021196:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761589.021208:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512866176:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761589.021213:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512866304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761589.021230:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761589.021233:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761589.021235:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761589.021238:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1563 00000100:00080000:2.0:1789761589.021243:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761589.021247:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761589.021253:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761589.021265:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512866432/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761589.021276:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761589.021280:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761589.021282:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761589.021287:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1563 00000100:00080000:2.0:1789761589.021292:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761589.021296:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789761589.021305:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:2.0:1789761589.021305:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761589.021324:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512866560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761589.021368:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761589/real 1789761589] req@ffff8dc00368dc00 x1876699512866176/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761589.021391:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512866176/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761589.021401:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761589.021405:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761589.021408:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512866176:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761589.021434:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512866304:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761589.021454:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761589.021464:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761589/real 1789761589] req@ffff8dc00368e680 x1876699512866304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761589.021472:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512866304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761589.021478:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761589.021480:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761589.021482:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512866304:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761589.021486:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512866432:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761589.021499:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761589.021506:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761589/real 1789761589] req@ffff8dc00368d180 x1876699512866432/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761589.021514:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512866432/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761589.021519:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761589.021522:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761589.021524:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512866432:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761589.021527:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512866560:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761589.021538:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761589.021544:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761589/real 1789761589] req@ffff8dc00368c700 x1876699512866560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761589.021552:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512866560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761589.021557:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761589.021559:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761589.021561:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512866560:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761590.045027:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761591.069005:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761592.093036:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761593.117039:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761594.141033:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761594.141041:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761594.141057:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761594.141070:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1568 00010000:00010000:2.0:1789761594.141071:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761594.141082:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761594.141090:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761594.141122:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512866688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761594.141143:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761594.141146:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761594.141148:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761594.141151:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1568 00000100:00080000:1.0:1789761594.141153:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761594.141156:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761594.141164:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761594.141173:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512866816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761594.141182:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761594.141185:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761594.141186:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761594.141188:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1568 00000100:00080000:1.0:1789761594.141191:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761594.141194:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761594.141201:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512866688:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1789761594.141202:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761594.141219:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512866944/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761594.141228:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761594.141231:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761594.141233:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761594.141236:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1568 00000100:00080000:1.0:1789761594.141238:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761594.141241:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761594.141245:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761594.141253:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512867072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761594.141302:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761594.141360:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761594/real 1789761594] req@ffff8dc005c96a00 x1876699512866688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761594.141378:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512866688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761594.141390:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761594.141394:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761594.141398:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512866688:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761594.141430:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512866816:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761594.141464:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761594.141475:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761594/real 1789761594] req@ffff8dc005c95500 x1876699512866816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761594.141484:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512866816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761594.141491:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761594.141494:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761594.141497:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512866816:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761594.141503:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512866944:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761594.141518:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761594.141527:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761594/real 1789761594] req@ffff8dc005c95f80 x1876699512866944/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761594.141539:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512866944/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761594.141552:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761594.141557:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761594.141559:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512866944:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761594.141564:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512867072:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761594.141582:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761594.141590:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761594/real 1789761594] req@ffff8dc005c94700 x1876699512867072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761594.141599:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512867072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761594.141605:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761594.141609:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761594.141612:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512867072:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761595.165084:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761596.189071:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761597.213075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761598.237077:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761599.261028:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761599.261043:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761599.261049:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761599.261064:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1573 00010000:00010000:1.0:1789761599.261080:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761599.261083:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761599.261095:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761599.261142:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512867200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761599.261170:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761599.261173:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761599.261175:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761599.261178:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1573 00000100:00080000:3.0:1789761599.261182:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761599.261185:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761599.261196:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761599.261209:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512867328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761599.261213:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512867200:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761599.261223:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761599.261227:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761599.261228:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761599.261231:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1573 00000100:00080000:3.0:1789761599.261234:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761599.261237:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761599.261241:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761599.261250:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512867456/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761599.261258:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761599.261261:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761599.261263:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761599.261265:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1573 00000100:00080000:3.0:1789761599.261268:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761599.261270:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761599.261273:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761599.261281:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512867584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761599.261293:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761599.261349:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761599/real 1789761599] req@ffff8dc0107ad180 x1876699512867200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761599.261365:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512867200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761599.261374:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761599.261378:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761599.261381:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512867200:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761599.261403:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512867328:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761599.261419:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761599.261428:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761599/real 1789761599] req@ffff8dc0107af480 x1876699512867328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761599.261437:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512867328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761599.261443:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761599.261445:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761599.261448:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512867328:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761599.261453:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512867456:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761599.261474:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761599.261488:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761599/real 1789761599] req@ffff8dc0107adf80 x1876699512867456/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761599.261501:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512867456/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761599.261509:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761599.261511:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761599.261514:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512867456:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761599.261518:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512867584:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761599.261535:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761599.261543:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761599/real 1789761599] req@ffff8dc0107ac380 x1876699512867584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761599.261551:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512867584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761599.261557:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761599.261559:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761599.261561:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512867584:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761600.285072:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761601.309074:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761602.333035:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761603.357041:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761604.380982:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761604.380993:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761604.380998:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761604.381009:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1578 00000100:00080000:3.0:1789761604.381020:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761604.381027:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761604.381037:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761604.381067:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512867712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761604.381093:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761604.381096:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761604.381097:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761604.381100:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1578 00000100:00080000:3.0:1789761604.381104:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761604.381107:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761604.381116:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761604.381127:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512867840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761604.381136:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512867712:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761604.381137:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761604.381145:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761604.381148:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761604.381151:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1578 00000100:00080000:3.0:1789761604.381153:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761604.381155:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761604.381158:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761604.381168:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512867968/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761604.381175:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761604.381178:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761604.381179:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761604.381182:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1578 00000100:00080000:3.0:1789761604.381184:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761604.381187:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761604.381189:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761604.381197:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512868096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761604.381209:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761604.381250:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761604/real 1789761604] req@ffff8dc0107af800 x1876699512867712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761604.381269:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512867712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761604.381280:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761604.381285:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761604.381289:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512867712:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761604.381310:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512867840:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761604.381329:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761604.381339:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761604/real 1789761604] req@ffff8dc0107ae680 x1876699512867840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761604.381348:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512867840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761604.381359:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761604.381361:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761604.381363:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512867840:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761604.381367:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512867968:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761604.381378:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761604.381384:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761604/real 1789761604] req@ffff8dc0107aea00 x1876699512867968/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761604.381392:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512867968/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761604.381397:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761604.381399:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761604.381401:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512867968:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761604.381405:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512868096:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761604.381414:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761604.381420:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761604/real 1789761604] req@ffff8dc0107ad880 x1876699512868096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761604.381427:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512868096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761604.381432:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761604.381433:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761604.381436:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512868096:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761605.405051:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761606.429076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761607.453084:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761608.477080:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761609.501018:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761609.501026:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761609.501029:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761609.501038:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1584 00000100:00080000:2.0:1789761609.501046:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761609.501054:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789761609.501055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789761609.501087:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512868224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761609.501107:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761609.501109:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761609.501111:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761609.501114:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1584 00000100:00080000:2.0:1789761609.501116:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761609.501119:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761609.501126:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761609.501135:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512868352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761609.501143:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761609.501146:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761609.501147:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761609.501152:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1584 00000100:00100000:0.0:1789761609.501154:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512868224:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789761609.501155:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761609.501160:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761609.501167:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761609.501174:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512868480/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761609.501182:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761609.501185:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761609.501186:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761609.501189:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1584 00000100:00080000:2.0:1789761609.501192:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761609.501194:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761609.501197:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761609.501204:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512868608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761609.501223:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761609.501260:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761609/real 1789761609] req@ffff8dc00368c700 x1876699512868224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761609.501274:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512868224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761609.501281:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761609.501284:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761609.501287:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512868224:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761609.501304:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512868352:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761609.501320:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761609.501328:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761609/real 1789761609] req@ffff8dc00368d180 x1876699512868352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761609.501336:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512868352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761609.501341:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761609.501343:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761609.501346:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512868352:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761609.501349:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512868480:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761609.501360:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761609.501376:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761609/real 1789761609] req@ffff8dc00368e680 x1876699512868480/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761609.501384:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512868480/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761609.501389:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761609.501391:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761609.501393:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512868480:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761609.501396:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512868608:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761609.501407:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761609.501413:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761609/real 1789761609] req@ffff8dc00368dc00 x1876699512868608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761609.501420:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512868608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761609.501425:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761609.501427:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761609.501428:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512868608:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761610.525070:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761611.549053:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761612.573069:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761613.597087:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761614.621035:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761614.621047:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761614.621051:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761614.621062:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1589 00000100:00080000:3.0:1789761614.621073:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1789761614.621075:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761614.621082:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761614.621122:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512868736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761614.621147:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761614.621151:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761614.621153:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761614.621157:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1589 00000100:00080000:3.0:1789761614.621162:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761614.621166:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761614.621176:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761614.621188:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512868864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761614.621197:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512868736:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761614.621202:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761614.621208:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761614.621210:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761614.621215:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1589 00000100:00080000:3.0:1789761614.621218:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761614.621222:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761614.621227:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761614.621241:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512868992/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761614.621251:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761614.621255:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761614.621257:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761614.621261:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1589 00000100:00080000:3.0:1789761614.621265:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761614.621269:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761614.621274:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761614.621288:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512869120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761614.621307:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512868864:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761614.621340:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512868992:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761614.621363:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512869120:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761615.645060:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761616.669005:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761617.693084:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789761617.693217:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1789761617.693283:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789761617.693304:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761617.693352:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761617.693362:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761617.693371:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:3.0:1789761617.693628:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761614/real 1789761617] req@ffff8dc0107af100 x1876699512868864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761617.693659:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512868864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761617.693673:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761617.693679:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761617.693683:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512868864:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761617.693693:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761614/real 1789761617] req@ffff8dc0107ad880 x1876699512868992/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761617.693706:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512868992/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761617.693715:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761617.693718:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761617.693722:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512868992:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761617.693730:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761614/real 1789761617] req@ffff8dc0107aea00 x1876699512869120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761617.693742:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512869120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761617.693751:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761617.693756:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761617.693760:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512869120:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761617.693767:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761614/real 1789761617] req@ffff8dc0107ace00 x1876699512868736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761617.693781:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512868736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761617.693790:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761617.693793:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761617.693797:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512868736:192.168.203.156@tcp:250:kworker.0 00010000:00010000:1.0:1789761618.717084:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761619.741032:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761619.741047:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761619.741052:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761619.741055:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761619.741064:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1594 00000100:00080000:1.0:1789761619.741076:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761619.741081:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761619.741110:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512869248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761619.741128:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761619.741131:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761619.741132:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761619.741135:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1594 00000100:00080000:1.0:1789761619.741138:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761619.741140:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761619.741147:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761619.741157:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512869376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761619.741166:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761619.741170:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1789761619.741170:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512869248:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761619.741174:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761619.741180:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1594 00000100:00080000:1.0:1789761619.741183:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761619.741185:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761619.741188:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761619.741198:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512869504/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761619.741209:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761619.741213:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761619.741215:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761619.741220:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1594 00000100:00080000:1.0:1789761619.741224:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761619.741228:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:3.0:1789761619.741234:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1789761619.741237:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761619.741251:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512869632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761619.741286:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761619/real 1789761619] req@ffff8dc005c97800 x1876699512869248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761619.741306:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512869248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761619.741317:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761619.741322:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761619.741326:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512869248:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761619.741352:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512869376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761619.741377:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761619.741391:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761619/real 1789761619] req@ffff8dc005c94a80 x1876699512869376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761619.741405:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512869376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761619.741415:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761619.741420:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761619.741423:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512869376:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761619.741429:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512869504:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761619.741452:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761619.741465:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761619/real 1789761619] req@ffff8dc005c97100 x1876699512869504/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761619.741477:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512869504/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761619.741485:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761619.741488:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761619.741497:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512869504:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761619.741502:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512869632:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761619.741523:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761619.741534:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761619/real 1789761619] req@ffff8dc005c95180 x1876699512869632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761619.741545:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512869632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761619.741553:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761619.741557:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761619.741560:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512869632:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761620.765053:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761621.789043:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761622.813056:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761623.837056:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761624.861037:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761624.861048:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761624.861053:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761624.861064:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1599 00000100:00080000:1.0:1789761624.861078:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761624.861084:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789761624.861103:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761624.861135:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512869760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761624.861157:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761624.861160:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761624.861161:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761624.861165:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1599 00000100:00080000:1.0:1789761624.861167:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761624.861171:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761624.861181:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761624.861189:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512869760:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761624.861196:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512869888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761624.861205:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761624.861207:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761624.861208:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761624.861211:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1599 00000100:00080000:1.0:1789761624.861213:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761624.861216:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761624.861220:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761624.861229:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512870016/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761624.861238:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761624.861242:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761624.861243:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761624.861246:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1599 00000100:00080000:1.0:1789761624.861248:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761624.861251:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761624.861254:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761624.861262:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512870144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761624.861266:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761624.861310:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761624/real 1789761624] req@ffff8dc005c95c00 x1876699512869760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761624.861332:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512869760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761624.861344:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761624.861349:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761624.861353:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512869760:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761624.861382:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512869888:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761624.861409:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761624.861424:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761624/real 1789761624] req@ffff8dc005c95180 x1876699512869888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761624.861439:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512869888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761624.861449:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761624.861452:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761624.861456:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512869888:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761624.861463:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512870016:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761624.861488:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761624.861502:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761624/real 1789761624] req@ffff8dc005c97100 x1876699512870016/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761624.861515:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512870016/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761624.861532:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761624.861536:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761624.861539:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512870016:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761624.861546:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512870144:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761624.861570:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761624.861583:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761624/real 1789761624] req@ffff8dc005c94a80 x1876699512870144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761624.861596:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512870144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761624.861605:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761624.861609:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761624.861613:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512870144:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761625.885060:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761626.909053:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761627.933088:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761628.957045:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761629.981016:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761629.981025:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761629.981029:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761629.981038:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1604 00000100:00080000:3.0:1789761629.981048:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761629.981056:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789761629.981067:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761629.981090:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512870272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761629.981124:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761629.981129:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761629.981131:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761629.981137:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1604 00000100:00080000:3.0:1789761629.981143:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761629.981149:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761629.981155:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512870272:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789761629.981158:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761629.981174:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512870400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761629.981186:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761629.981189:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761629.981191:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761629.981195:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1604 00000100:00080000:3.0:1789761629.981198:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761629.981202:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761629.981214:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789761629.981225:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761629.981230:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512870528/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761629.981264:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761629/real 1789761629] req@ffff8dc0107ace00 x1876699512870272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761629.981284:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512870272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761629.981303:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761629.981308:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1789761629.981309:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789761629.981312:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512870272:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761629.981312:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761629.981314:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761629.981318:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1604 00000100:00080000:3.0:1789761629.981322:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761629.981326:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761629.981331:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761629.981334:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512870400:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761629.981356:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761629.981371:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761629/real 1789761629] req@ffff8dc0107aea00 x1876699512870400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761629.981385:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512870400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761629.981394:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761629.981398:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761629.981401:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512870400:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761629.981407:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512870528:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761629.981427:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761629.981438:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761629/real 1789761629] req@ffff8dc0107ad880 x1876699512870528/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761629.981451:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512870528/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761629.981459:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761629.981462:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761629.981466:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512870528:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761629.981470:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512870656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761629.981498:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512870656:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761629.981525:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761629.981542:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761629/real 1789761629] req@ffff8dc0107af100 x1876699512870656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761629.981555:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512870656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761629.981571:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761629.981574:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761629.981578:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512870656:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761631.005087:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761632.029040:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761633.053027:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761634.077105:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761635.101001:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1789761635.101005:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761635.101011:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761635.101015:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761635.101023:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1609 00000100:00080000:1.0:1789761635.101030:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761635.101035:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761635.101066:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512870784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761635.101089:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761635.101094:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761635.101096:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761635.101102:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1609 00000100:00080000:1.0:1789761635.101107:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761635.101112:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761635.101123:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761635.101124:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512870784:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761635.101143:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512870912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761635.101153:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761635.101157:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761635.101158:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761635.101161:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1609 00000100:00080000:1.0:1789761635.101164:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761635.101168:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761635.101172:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761635.101181:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512871040/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761635.101189:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761635.101192:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761635.101195:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789761635.101196:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761635.101200:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1609 00000100:00080000:1.0:1789761635.101204:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761635.101207:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761635.101211:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761635.101219:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512871168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761635.101242:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761635/real 1789761635] req@ffff8dc005c97800 x1876699512870784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761635.101256:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512870784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761635.101263:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761635.101266:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761635.101269:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512870784:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761635.101287:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512870912:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761635.101304:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761635.101312:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761635/real 1789761635] req@ffff8dc005c94700 x1876699512870912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761635.101321:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512870912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761635.101340:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761635.101342:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761635.101345:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512870912:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761635.101349:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512871040:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761635.101362:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761635.101369:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761635/real 1789761635] req@ffff8dc005c95f80 x1876699512871040/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761635.101377:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512871040/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761635.101382:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761635.101384:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761635.101386:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512871040:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761635.101390:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512871168:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761635.101400:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761635.101406:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761635/real 1789761635] req@ffff8dc005c95500 x1876699512871168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761635.101413:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512871168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761635.101418:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761635.101420:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761635.101422:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512871168:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761636.125096:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761637.149036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761638.173094:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761639.197069:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761640.221017:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761640.221028:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761640.221033:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761640.221048:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1614 00000100:00080000:1.0:1789761640.221063:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761640.221072:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761640.221124:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512871296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761640.221157:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761640.221162:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761640.221165:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761640.221170:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1614 00000100:00080000:1.0:1789761640.221175:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761640.221179:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761640.221186:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512871296:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1789761640.221191:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761640.221212:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512871424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761640.221224:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761640.221228:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761640.221230:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761640.221234:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1614 00000800:00000100:0.0:1789761640.221268:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00010000:00010000:3.0:1789761640.221306:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1789761640.221326:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761640/real 1789761640] req@ffff8dc005c96a00 x1876699512871296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761640.221336:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761640.221345:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761640.221351:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512871296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789761640.221355:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00080000:0.0:1789761640.221373:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761640.221382:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761640.221382:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512871552/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761640.221392:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512871296:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761640.221412:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761640.221416:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761640.221418:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761640.221424:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1614 00000100:00080000:1.0:1789761640.221429:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761640.221432:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512871424:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1789761640.221437:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761640.221446:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761640.221462:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512871680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761640.221472:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761640.221496:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761640/real 1789761640] req@ffff8dc005c94380 x1876699512871424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761640.221512:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512871424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761640.221522:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761640.221527:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761640.221530:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512871424:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761640.221537:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512871552:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761640.221555:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761640.221564:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761640/real 1789761640] req@ffff8dc005c97b80 x1876699512871552/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761640.221572:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512871552/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761640.221578:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761640.221581:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761640.221583:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512871552:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761640.221593:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512871680:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761640.221605:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761640.221611:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761640/real 1789761640] req@ffff8dc005c96300 x1876699512871680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761640.221619:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512871680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761640.221625:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761640.221627:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761640.221631:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512871680:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761641.245013:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761642.269036:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761643.293054:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761644.317132:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761645.341178:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761645.341204:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761645.341212:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761645.341217:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761645.341228:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1619 00000100:00080000:3.0:1789761645.341238:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761645.341245:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761645.341279:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512871808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761645.341305:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761645.341308:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761645.341310:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761645.341315:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1619 00000100:00080000:3.0:1789761645.341318:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761645.341323:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761645.341332:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761645.341346:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512871808:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761645.341347:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512871936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761645.341364:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761645.341368:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761645.341370:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761645.341379:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1619 00000100:00080000:3.0:1789761645.341382:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761645.341385:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761645.341391:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761645.341407:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512872064/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761645.341408:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761645.341422:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761645.341427:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761645.341429:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761645.341435:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1619 00000100:00080000:3.0:1789761645.341440:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761645.341443:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761645/real 1789761645] req@ffff8dc0107ae680 x1876699512871808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761645.341448:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761645.341455:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761645.341470:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512872192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761645.341474:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512871808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761645.341502:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761645.341506:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761645.341510:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512871808:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761645.341528:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512871936:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761645.341544:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761645.341553:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761645/real 1789761645] req@ffff8dc0107af800 x1876699512871936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761645.341561:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512871936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761645.341567:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761645.341569:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761645.341571:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512871936:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761645.341575:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512872064:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761645.341586:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761645.341592:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761645/real 1789761645] req@ffff8dc0107ac380 x1876699512872064/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761645.341605:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512872064/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761645.341610:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761645.341612:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761645.341614:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512872064:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761645.341617:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512872192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761645.341627:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761645.341633:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761645/real 1789761645] req@ffff8dc0107adf80 x1876699512872192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761645.341640:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512872192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761645.341645:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761645.341647:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761645.341649:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512872192:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761646.365033:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761647.389077:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761648.413036:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761649.437018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761650.460998:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761650.461008:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761650.461013:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761650.461024:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1625 00010000:00010000:3.0:1789761650.461031:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761650.461034:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761650.461041:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761650.461073:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512872320/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761650.461093:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761650.461097:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761650.461100:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761650.461104:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1625 00000100:00080000:1.0:1789761650.461108:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761650.461112:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761650.461122:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761650.461127:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512872320:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761650.461133:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512872448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761650.461153:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761650.461157:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761650.461159:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761650.461162:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1625 00000100:00080000:1.0:1789761650.461172:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761650.461176:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761650.461181:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761650.461184:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761650.461191:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512872576/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761650.461200:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761650.461203:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761650.461205:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761650.461209:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1625 00000100:00080000:1.0:1789761650.461213:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761650.461216:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761650.461221:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761650.461231:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761650/real 1789761650] req@ffff8dc005c94e00 x1876699512872320/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761650.461231:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512872704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761650.461250:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512872320/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761650.461268:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761650.461273:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761650.461278:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512872320:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761650.461299:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512872448:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761650.461325:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761650.461339:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761650/real 1789761650] req@ffff8dc005c97480 x1876699512872448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761650.461353:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512872448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761650.461362:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761650.461366:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761650.461369:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512872448:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761650.461376:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512872576:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761650.461398:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761650.461407:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761650/real 1789761650] req@ffff8dc005c96680 x1876699512872576/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761650.461415:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512872576/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761650.461421:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761650.461423:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761650.461425:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512872576:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761650.461429:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512872704:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761650.461446:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761650.461454:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761650/real 1789761650] req@ffff8dc005c95880 x1876699512872704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761650.461462:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512872704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761650.461467:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761650.461469:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761650.461471:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512872704:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761651.485109:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761652.509078:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761653.533049:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761654.557069:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761655.581047:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761655.581058:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:1.0:1789761655.581061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761655.581063:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761655.581077:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1630 00000100:00080000:2.0:1789761655.581090:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761655.581097:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761655.581140:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512872832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761655.581165:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761655.581168:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761655.581170:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761655.581177:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1630 00000100:00080000:2.0:1789761655.581181:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761655.581184:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761655.581194:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761655.581204:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512872960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761655.581215:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512872832:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789761655.581219:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761655.581229:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761655.581233:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761655.581239:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1630 00000100:00080000:2.0:1789761655.581242:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761655.581244:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761655.581247:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761655.581256:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512873088/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761655.581264:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761655.581267:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761655.581268:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761655.581272:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1630 00000100:00080000:2.0:1789761655.581274:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761655.581277:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761655.581280:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761655.581291:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512873216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761655.581305:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761655.581347:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761655/real 1789761655] req@ffff8dc00368f800 x1876699512872832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761655.581364:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512872832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761655.581372:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761655.581375:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761655.581378:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512872832:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761655.581397:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512872960:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761655.581414:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761655.581423:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761655/real 1789761655] req@ffff8dc00368e300 x1876699512872960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761655.581432:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512872960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761655.581437:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761655.581440:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761655.581442:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512872960:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761655.581446:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512873088:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761655.581457:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761655.581463:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761655/real 1789761655] req@ffff8dc00368ed80 x1876699512873088/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761655.581470:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512873088/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761655.581475:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761655.581477:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761655.581479:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512873088:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761655.581482:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512873216:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761655.581493:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761655.581498:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761655/real 1789761655] req@ffff8dc00368df80 x1876699512873216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761655.581505:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512873216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761655.581515:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761655.581517:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761655.581519:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512873216:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761656.605018:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761657.629090:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761658.653043:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761659.677062:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761660.701024:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761660.701036:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761660.701041:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1789761660.701049:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761660.701054:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1635 00000100:00080000:3.0:1789761660.701065:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761660.701072:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761660.701106:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512873344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761660.701131:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761660.701136:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761660.701138:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761660.701142:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1635 00000100:00080000:3.0:1789761660.701146:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761660.701151:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761660.701160:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761660.701172:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512873344:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761660.701173:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512873472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761660.701188:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761660.701190:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761660.701192:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761660.701195:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1635 00000100:00080000:3.0:1789761660.701198:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761660.701200:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761660.701204:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761660.701213:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512873600/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761660.701225:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761660.701227:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761660.701228:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761660.701233:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1635 00000800:00000100:0.0:1789761660.701234:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761660.701237:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761660.701241:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761660.701245:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761660.701253:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512873728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761660.701276:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761660/real 1789761660] req@ffff8dc0107af480 x1876699512873344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761660.701293:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512873344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761660.701301:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761660.701304:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761660.701307:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512873344:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761660.701324:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512873472:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761660.701344:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761660.701353:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761660/real 1789761660] req@ffff8dc0107ad180 x1876699512873472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761660.701361:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512873472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761660.701367:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761660.701369:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761660.701371:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512873472:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761660.701376:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512873600:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761660.701388:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761660.701394:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761660/real 1789761660] req@ffff8dc0107aed80 x1876699512873600/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761660.701402:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512873600/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761660.701407:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761660.701409:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761660.701412:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512873600:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761660.701415:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512873728:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761660.701425:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761660.701431:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761660/real 1789761660] req@ffff8dc0107ad500 x1876699512873728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761660.701438:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512873728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761660.701443:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761660.701445:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761660.701447:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512873728:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761661.725073:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761662.749045:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761663.773140:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761664.797062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761665.821011:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761665.821019:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761665.821023:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761665.821031:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1640 00000100:00080000:0.0:1789761665.821039:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761665.821045:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761665.821075:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148a80 x1876699512873856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761665.821100:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761665.821103:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761665.821105:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761665.821110:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1640 00000100:00080000:0.0:1789761665.821114:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761665.821118:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761665.821127:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761665.821145:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514aa00 x1876699512873984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761665.821152:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512873856:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789761665.821162:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761665.821167:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761665.821169:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761665.821174:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1640 00000100:00080000:0.0:1789761665.821178:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761665.821183:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761665.821189:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761665.821201:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148700 x1876699512874112/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761665.821211:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1789761665.821223:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761665.821227:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761665.821229:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761665.821234:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1640 00000100:00080000:0.0:1789761665.821238:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761665.821242:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761665.821247:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761665.821258:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514ad80 x1876699512874240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761665.821264:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761665/real 1789761665] req@ffff8dc005148a80 x1876699512873856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761665.821292:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148a80 x1876699512873856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761665.821305:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761665.821310:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761665.821317:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512873856:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761665.821345:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512873984:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761665.821377:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:3.0:1789761665.821378:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761665.821401:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761665/real 1789761665] req@ffff8dc00514aa00 x1876699512873984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761665.821417:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514aa00 x1876699512873984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761665.821427:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761665.821431:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761665.821437:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512873984:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761665.821445:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512874112:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761665.821480:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761665.821496:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761665/real 1789761665] req@ffff8dc005148700 x1876699512874112/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761665.821507:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148700 x1876699512874112/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761665.821530:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761665.821535:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761665.821538:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512874112:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761665.821550:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512874240:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761665.821568:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761665.821579:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761665/real 1789761665] req@ffff8dc00514ad80 x1876699512874240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761665.821589:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514ad80 x1876699512874240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761665.821599:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761665.821602:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761665.821658:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512874240:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761666.845046:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761667.869049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761668.893101:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761669.917069:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761670.941006:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761670.941015:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761670.941018:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761670.941027:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1645 00000100:00080000:0.0:1789761670.941035:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761670.941041:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761670.941072:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148000 x1876699512874368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761670.941093:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761670.941096:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761670.941097:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761670.941100:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1645 00000100:00080000:0.0:1789761670.941102:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761670.941105:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761670.941114:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761670.941127:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512874368:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761670.941137:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148380 x1876699512874496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761670.941151:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761670.941154:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761670.941157:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761670.941161:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1645 00000100:00080000:0.0:1789761670.941166:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761670.941171:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761670.941177:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1789761670.941185:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761670.941192:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514bb80 x1876699512874624/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761670.941203:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761670.941207:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761670.941209:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761670.941214:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1645 00000100:00080000:0.0:1789761670.941218:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761670.941223:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761670.941228:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761670/real 1789761670] req@ffff8dc005148000 x1876699512874368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789761670.941230:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761670.941247:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148000 x1876699512874368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761670.941251:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b480 x1876699512874752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761670.941259:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761670.941267:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761670.941272:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512874368:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761670.941290:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512874496:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761670.941312:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761670.941327:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761670/real 1789761670] req@ffff8dc005148380 x1876699512874496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761670.941341:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148380 x1876699512874496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761670.941349:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761670.941353:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761670.941357:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512874496:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761670.941368:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761670.941368:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512874624:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761670.941397:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761670.941406:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761670/real 1789761670] req@ffff8dc00514bb80 x1876699512874624/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761670.941414:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514bb80 x1876699512874624/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761670.941420:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761670.941422:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761670.941424:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512874624:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761670.941428:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512874752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761670.941438:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761670.941444:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761670/real 1789761670] req@ffff8dc00514b480 x1876699512874752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761670.941451:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b480 x1876699512874752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761670.941457:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761670.941458:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761670.941460:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512874752:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761671.965090:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761672.989173:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761674.013067:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761675.037052:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761676.061007:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761676.061018:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761676.061022:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761676.061034:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1650 00000100:00080000:0.0:1789761676.061048:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761676.061057:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761676.061110:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149c00 x1876699512874880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761676.061135:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761676.061138:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761676.061140:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761676.061143:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1650 00000100:00080000:0.0:1789761676.061147:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761676.061151:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761676.061162:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761676.061172:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512874880:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761676.061178:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149180 x1876699512875008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761676.061192:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761676.061196:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761676.061198:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761676.061203:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1650 00000100:00080000:0.0:1789761676.061207:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761676.061212:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761676.061217:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761676.061227:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149f80 x1876699512875136/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761676.061236:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761676.061239:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761676.061240:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789761676.061244:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1789761676.061248:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1650 00000100:00080000:0.0:1789761676.061253:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761676.061255:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761676.061259:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761676.061270:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b800 x1876699512875264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761676.061312:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761676/real 1789761676] req@ffff8dc005149c00 x1876699512874880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:2.0:1789761676.061332:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761676.061339:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149c00 x1876699512874880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761676.061353:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761676.061357:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761676.061361:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512874880:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761676.061387:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512875008:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761676.061413:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761676.061428:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761676/real 1789761676] req@ffff8dc005149180 x1876699512875008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761676.061455:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149180 x1876699512875008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761676.061462:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761676.061464:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761676.061467:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512875008:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761676.061472:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512875136:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761676.061490:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761676.061499:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761676/real 1789761676] req@ffff8dc005149f80 x1876699512875136/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761676.061507:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149f80 x1876699512875136/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761676.061518:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761676.061520:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761676.061522:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512875136:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761676.061526:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512875264:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761676.061537:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761676.061543:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761676/real 1789761676] req@ffff8dc00514b800 x1876699512875264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761676.061550:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b800 x1876699512875264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761676.061555:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761676.061557:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761676.061559:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512875264:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761677.085076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761678.109046:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761679.133083:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761680.157027:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761681.181023:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761681.181034:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761681.181039:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761681.181050:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1655 00000100:00080000:2.0:1789761681.181061:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761681.181071:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789761681.181091:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789761681.181107:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512875392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761681.181131:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761681.181135:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761681.181137:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761681.181142:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1655 00000100:00080000:2.0:1789761681.181147:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761681.181152:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761681.181162:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761681.181165:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512875392:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761681.181178:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512875520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761681.181187:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761681.181189:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761681.181191:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761681.181194:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1655 00000100:00080000:2.0:1789761681.181196:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761681.181200:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761681.181204:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761681.181212:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512875648/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761681.181219:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761681.181222:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761681.181223:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761681.181226:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1655 00000100:00080000:2.0:1789761681.181228:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761681.181230:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761681.181234:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761681.181242:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512875776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761681.181269:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512875520:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761681.181296:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512875648:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761681.181310:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512875776:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761682.205034:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761683.229095:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761684.253149:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789761684.253246:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1789761684.253305:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789761684.253326:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761684.253388:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761684.253405:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761684.253417:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:3.0:1789761684.253441:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761681/real 1789761684] req@ffff8dc00368ca80 x1876699512875520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761684.253474:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512875520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761684.253503:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761684.253510:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761684.253517:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512875520:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761684.253525:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761681/real 1789761684] req@ffff8dc00368c000 x1876699512875648/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761684.253539:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512875648/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761684.253549:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761684.253553:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761684.253557:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512875648:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761684.253564:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761681/real 1789761684] req@ffff8dc00368ea00 x1876699512875776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761684.253578:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512875776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761684.253586:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761684.253590:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761684.253593:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512875776:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761684.253600:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761681/real 1789761684] req@ffff8dc00368ce00 x1876699512875392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761684.253613:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512875392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761684.253620:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761684.253623:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761684.253626:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512875392:192.168.203.156@tcp:250:kworker.0 00010000:00010000:2.0:1789761685.277051:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761686.301055:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761686.301065:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761686.301071:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761686.301082:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1660 00010000:00010000:0.0:1789761686.301093:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761686.301096:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761686.301108:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761686.301143:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512875904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761686.301172:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761686.301177:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761686.301179:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761686.301184:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1660 00000100:00080000:2.0:1789761686.301188:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761686.301193:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761686.301204:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761686.301221:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512876032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761686.301231:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512875904:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789761686.301248:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761686.301252:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761686.301255:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761686.301262:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1660 00000100:00080000:2.0:1789761686.301268:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761686.301272:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761686.301278:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761686.301298:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512876160/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761686.301311:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761686.301315:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761686.301317:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761686.301321:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1660 00000100:00080000:2.0:1789761686.301327:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789761686.301334:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789761686.301336:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761686.301347:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761686.301381:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512876288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761686.301421:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761686/real 1789761686] req@ffff8dc00368f480 x1876699512875904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761686.301446:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512875904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761686.301460:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761686.301465:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761686.301471:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512875904:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761686.301506:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512876032:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761686.301536:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761686.301552:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761686/real 1789761686] req@ffff8dc00368fb80 x1876699512876032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761686.301566:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512876032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761686.301575:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761686.301578:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761686.301582:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512876032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761686.301590:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512876160:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761686.301614:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761686.301626:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761686/real 1789761686] req@ffff8dc00368d880 x1876699512876160/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761686.301637:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512876160/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761686.301646:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761686.301649:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761686.301652:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512876160:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761686.301660:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512876288:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761686.301682:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761686.301694:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761686/real 1789761686] req@ffff8dc00368f100 x1876699512876288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761686.301706:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512876288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761686.301715:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761686.301719:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761686.301723:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512876288:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761687.325113:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761688.349059:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761689.373070:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761690.397043:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761691.420988:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761691.420999:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761691.421004:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761691.421016:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1666 00000100:00080000:0.0:1789761691.421025:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761691.421034:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761691.421076:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149880 x1876699512876416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761691.421098:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761691.421101:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761691.421102:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761691.421105:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1666 00000100:00080000:0.0:1789761691.421108:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761691.421111:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761691.421119:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761691.421128:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514a300 x1876699512876544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761691.421136:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512876416:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789761691.421139:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761691.421143:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761691.421145:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761691.421153:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1666 00000100:00080000:0.0:1789761691.421156:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761691.421158:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761691.421162:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761691.421171:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148e00 x1876699512876672/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761691.421183:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761691.421186:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761691.421187:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761691.421189:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1666 00000100:00080000:0.0:1789761691.421193:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789761691.421193:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1789761691.421199:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761691.421203:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761691.421212:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b100 x1876699512876800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761691.421233:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761691/real 1789761691] req@ffff8dc005149880 x1876699512876416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761691.421248:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149880 x1876699512876416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761691.421259:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761691.421264:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761691.421270:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512876416:192.168.203.156@tcp:250:kworker.0 00010000:00010000:2.0:1789761691.421277:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761691.421297:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512876544:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761691.421342:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761691.421353:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761691/real 1789761691] req@ffff8dc00514a300 x1876699512876544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761691.421366:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514a300 x1876699512876544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761691.421375:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761691.421378:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761691.421381:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512876544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761691.421388:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512876672:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761691.421410:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761691.421421:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761691/real 1789761691] req@ffff8dc005148e00 x1876699512876672/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761691.421430:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148e00 x1876699512876672/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761691.421440:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761691.421442:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761691.421444:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512876672:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761691.421448:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512876800:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761691.421462:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761691.421469:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761691/real 1789761691] req@ffff8dc00514b100 x1876699512876800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761691.421476:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b100 x1876699512876800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761691.421481:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761691.421483:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761691.421485:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512876800:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761692.445012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761693.469032:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761694.493025:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761695.517006:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761696.541020:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761696.541028:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761696.541032:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761696.541041:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1671 00010000:00010000:3.0:1789761696.541047:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761696.541051:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761696.541058:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761696.541088:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96d80 x1876699512876928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761696.541110:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761696.541113:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761696.541115:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761696.541118:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1671 00000100:00080000:1.0:1789761696.541121:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761696.541125:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761696.541134:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761696.541142:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512876928:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761696.541147:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512877056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761696.541157:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761696.541159:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761696.541161:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761696.541164:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1671 00000100:00080000:1.0:1789761696.541167:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761696.541169:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761696.541173:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761696.541182:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512877184/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761696.541190:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761696.541194:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761696.541196:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761696.541200:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1671 00000800:00000100:0.0:1789761696.541205:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761696.541207:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761696.541215:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761696.541222:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761696.541236:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512877312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761696.541240:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761696/real 1789761696] req@ffff8dc005c96d80 x1876699512876928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761696.541268:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96d80 x1876699512876928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761696.541293:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761696.541298:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761696.541302:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512876928:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761696.541322:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512877056:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761696.541340:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761696.541349:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761696/real 1789761696] req@ffff8dc005c95880 x1876699512877056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761696.541358:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512877056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761696.541364:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761696.541366:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761696.541368:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512877056:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761696.541372:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512877184:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761696.541383:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761696.541389:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761696/real 1789761696] req@ffff8dc005c96680 x1876699512877184/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761696.541396:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512877184/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761696.541402:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761696.541406:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761696.541409:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512877184:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761696.541415:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512877312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761696.541434:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761696.541446:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761696/real 1789761696] req@ffff8dc005c97480 x1876699512877312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761696.541459:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512877312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761696.541468:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761696.541472:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761696.541475:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512877312:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761697.564996:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761698.589012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761699.613019:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761700.637047:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761701.661221:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761701.661227:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761701.661231:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761701.661251:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1676 00000100:00080000:1.0:1789761701.661260:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761701.661268:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761701.661302:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512877440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761701.661333:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761701.661338:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761701.661340:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761701.661344:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1676 00000100:00080000:1.0:1789761701.661347:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761701.661351:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761701.661358:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761701.661369:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512877568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761701.661376:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761701.661379:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761701.661380:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761701.661383:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1676 00000100:00080000:1.0:1789761701.661386:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761701.661388:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761701.661391:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761701.661398:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512877696/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761701.661404:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761701.661406:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761701.661407:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761701.661415:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1676 00000100:00080000:1.0:1789761701.661417:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761701.661420:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761701.661423:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761701.661429:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512877824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761701.661493:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512877440:192.168.203.156@tcp:250:kworker.0 00010000:00010000:0.0:1789761701.661564:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:3.0:1789761701.661591:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761701.661634:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761701/real 1789761701] req@ffff8dc005c94e00 x1876699512877440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761701.661651:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512877440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761701.661661:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761701.661666:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761701.661672:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512877440:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761701.661680:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512877568:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761701.661703:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761701.661717:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761701/real 1789761701] req@ffff8dc005c96300 x1876699512877568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761701.661726:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512877568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761701.661732:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761701.661734:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761701.661737:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512877568:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761701.661741:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512877696:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761701.661764:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761701.661772:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761701/real 1789761701] req@ffff8dc005c97b80 x1876699512877696/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761701.661780:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512877696/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761701.661785:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761701.661787:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761701.661789:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512877696:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761701.661793:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512877824:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761701.661805:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761701.661811:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761701/real 1789761701] req@ffff8dc005c94380 x1876699512877824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761701.661822:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512877824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761701.661831:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761701.661834:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761701.661837:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512877824:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761702.685049:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761703.709020:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761704.733017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761705.757087:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761706.781020:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761706.781028:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761706.781032:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761706.781042:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1681 00000100:00080000:1.0:1789761706.781053:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761706.781060:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761706.781092:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512877952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1789761706.781096:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761706.781116:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761706.781119:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761706.781121:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761706.781124:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1681 00000100:00080000:1.0:1789761706.781126:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761706.781129:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761706.781136:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761706.781146:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512878080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761706.781155:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761706.781157:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761706.781159:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761706.781162:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1681 00000100:00080000:1.0:1789761706.781165:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761706.781173:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761706.781175:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512877952:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1789761706.781177:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761706.781195:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512878208/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761706.781215:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761706.781219:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761706.781221:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761706.781225:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1681 00000100:00080000:1.0:1789761706.781229:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761706.781232:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761706.781236:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761706.781246:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512878336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761706.781292:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761706.781341:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761706/real 1789761706] req@ffff8dc005c96a00 x1876699512877952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761706.781364:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512877952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761706.781378:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761706.781383:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761706.781386:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512877952:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761706.781413:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512878080:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761706.781433:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761706.781444:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761706/real 1789761706] req@ffff8dc005c95500 x1876699512878080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761706.781459:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512878080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761706.781465:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761706.781467:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761706.781469:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512878080:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761706.781474:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512878208:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761706.781487:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761706.781494:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761706/real 1789761706] req@ffff8dc005c95f80 x1876699512878208/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761706.781502:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512878208/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761706.781508:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761706.781510:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761706.781512:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512878208:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761706.781516:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512878336:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761706.781527:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761706.781533:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761706/real 1789761706] req@ffff8dc005c94700 x1876699512878336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761706.781541:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512878336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761706.781551:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761706.781553:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761706.781556:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512878336:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761707.805050:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761708.829031:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761709.853061:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761710.877006:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761711.901074:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761711.901082:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761711.901086:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761711.901094:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1686 00000100:00080000:0.0:1789761711.901101:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761711.901107:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761711.901141:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149500 x1876699512878464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761711.901210:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512878464:192.168.203.156@tcp:250:kworker.0 00000800:00000100:3.0:1789761711.901266:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1789761711.901298:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761711.901303:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761711.901305:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761711.901310:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1686 00000100:00080000:0.0:1789761711.901314:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761711.901319:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761711.901329:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761711.901345:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b100 x1876699512878592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761711.901407:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761711/real 1789761711] req@ffff8dc005149500 x1876699512878464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761711.901425:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149500 x1876699512878464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761711.901437:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761711.901441:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761711.901445:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512878464:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761711.901466:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512878592:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761711.901487:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761711.901500:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761711/real 1789761711] req@ffff8dc00514b100 x1876699512878592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761711.901510:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b100 x1876699512878592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761711.901516:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761711.901518:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761711.901521:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512878592:192.168.203.156@tcp:38:kworker.0 00000100:00080000:0.0:1789761711.901567:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761711.901571:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761711.901573:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761711.901578:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1686 00000100:00080000:0.0:1789761711.901582:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761711.901587:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761711.901593:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761711.901607:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148e00 x1876699512878720/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761711.901619:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761711.901623:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761711.901625:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761711.901628:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1686 00000100:00080000:0.0:1789761711.901632:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761711.901636:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761711.901641:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761711.901650:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514a300 x1876699512878848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1789761711.901738:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761711.901758:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512878720:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761711.901785:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761711.901799:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761711/real 1789761711] req@ffff8dc005148e00 x1876699512878720/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761711.901812:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148e00 x1876699512878720/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761711.901821:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761711.901824:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761711.901828:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512878720:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761711.901835:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512878848:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761711.901852:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761711.901862:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761711/real 1789761711] req@ffff8dc00514a300 x1876699512878848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761711.901873:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514a300 x1876699512878848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761711.901882:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761711.901885:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761711.901888:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512878848:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761712.925038:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761713.949055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761714.973017:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761715.997076:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761717.020989:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761717.020998:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761717.021003:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761717.021014:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1691 00010000:00010000:0.0:1789761717.021024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761717.021025:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761717.021034:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761717.021066:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512878976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761717.021088:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761717.021091:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761717.021093:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761717.021096:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1691 00000100:00080000:2.0:1789761717.021099:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761717.021103:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761717.021111:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761717.021119:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512879104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761717.021141:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761717.021145:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761717.021146:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761717.021151:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1691 00000100:00100000:3.0:1789761717.021153:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512878976:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789761717.021154:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761717.021161:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761717.021167:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761717.021177:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512879232/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761717.021184:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761717.021186:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761717.021188:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761717.021191:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1691 00000100:00080000:2.0:1789761717.021193:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761717.021195:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761717.021198:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761717.021204:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512879360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761717.021224:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761717.021263:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761717/real 1789761717] req@ffff8dc00368f100 x1876699512878976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761717.021280:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512878976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761717.021291:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761717.021295:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761717.021299:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512878976:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761717.021319:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512879104:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761717.021340:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761717.021362:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761717/real 1789761717] req@ffff8dc00368d880 x1876699512879104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761717.021375:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512879104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761717.021383:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761717.021386:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761717.021390:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512879104:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761717.021396:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512879232:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761717.021414:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761717.021425:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761717/real 1789761717] req@ffff8dc00368fb80 x1876699512879232/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761717.021437:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512879232/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761717.021451:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761717.021455:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761717.021458:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512879232:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761717.021464:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512879360:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761717.021480:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761717.021491:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761717/real 1789761717] req@ffff8dc00368f480 x1876699512879360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761717.021502:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512879360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761717.021510:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761717.021514:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761717.021517:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512879360:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761718.045071:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761719.069107:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761720.092973:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761721.117016:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761722.141048:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761722.141057:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761722.141072:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761722.141080:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1696 00000100:00080000:2.0:1789761722.141105:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761722.141112:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789761722.141130:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789761722.141161:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512879488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761722.141183:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761722.141186:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761722.141187:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761722.141191:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1696 00000100:00080000:2.0:1789761722.141193:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761722.141196:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761722.141204:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761722.141213:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512879616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761722.141221:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761722.141223:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761722.141224:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761722.141226:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1696 00000100:00080000:2.0:1789761722.141228:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761722.141230:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761722.141235:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761722.141242:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512879488:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761722.141248:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512879744/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761722.141263:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761722.141265:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761722.141266:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761722.141270:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1696 00000100:00080000:2.0:1789761722.141272:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761722.141274:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761722.141277:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761722.141285:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512879872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761722.141348:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761722.141420:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761722/real 1789761722] req@ffff8dc00368ce00 x1876699512879488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761722.141437:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512879488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761722.141447:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761722.141451:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761722.141455:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512879488:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761722.141501:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512879616:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761722.141530:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761722.141548:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761722/real 1789761722] req@ffff8dc00368ea00 x1876699512879616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761722.141566:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512879616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761722.141578:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761722.141582:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761722.141602:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512879616:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761722.141607:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512879744:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761722.141631:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761722.141643:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761722/real 1789761722] req@ffff8dc00368c000 x1876699512879744/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761722.141657:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512879744/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761722.141666:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761722.141670:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761722.141674:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512879744:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761722.141681:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512879872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761722.141703:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761722.141715:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761722/real 1789761722] req@ffff8dc00368ca80 x1876699512879872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761722.141724:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512879872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761722.141730:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761722.141732:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761722.141734:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512879872:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761723.165025:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761724.189084:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761725.212991:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761726.237036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761727.261024:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761727.261034:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761727.261039:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761727.261060:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1701 00000100:00080000:3.0:1789761727.261072:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761727.261080:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761727.261118:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512880000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1789761727.261126:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761727.261152:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761727.261157:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761727.261159:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761727.261165:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1701 00000100:00080000:3.0:1789761727.261171:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761727.261176:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761727.261189:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761727.261197:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512880000:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761727.261212:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512880128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761727.261228:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761727.261232:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761727.261234:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761727.261239:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1701 00000100:00080000:3.0:1789761727.261243:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761727.261248:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761727.261254:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761727.261269:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512880256/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761727.261281:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761727.261285:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1789761727.261287:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761727.261289:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761727.261298:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1701 00000100:00080000:3.0:1789761727.261305:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761727.261309:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761727.261317:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761727.261334:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512880384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761727.261336:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761727/real 1789761727] req@ffff8dc0107ac000 x1876699512880000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761727.261412:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512880000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761727.261423:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761727.261428:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761727.261432:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512880000:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761727.261456:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512880128:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761727.261474:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761727.261483:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761727/real 1789761727] req@ffff8dc0107ac700 x1876699512880128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761727.261491:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512880128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761727.261497:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761727.261500:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761727.261503:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512880128:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761727.261507:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512880256:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761727.261519:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761727.261525:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761727/real 1789761727] req@ffff8dc0107ae300 x1876699512880256/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761727.261545:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512880256/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761727.261551:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761727.261554:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761727.261556:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512880256:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761727.261559:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512880384:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761727.261572:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761727.261579:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761727/real 1789761727] req@ffff8dc0107aca80 x1876699512880384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761727.261587:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512880384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761727.261592:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761727.261594:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761727.261596:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512880384:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761728.285040:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761729.309014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761730.333074:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761731.357054:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761732.380989:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761732.380999:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:0.0:1789761732.381001:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761732.381004:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761732.381030:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1706 00000100:00080000:3.0:1789761732.381040:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761732.381047:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761732.381084:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699512880512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761732.381111:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761732.381115:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761732.381117:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761732.381121:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1706 00000100:00080000:3.0:1789761732.381126:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761732.381131:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761732.381141:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761732.381149:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512880512:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761732.381158:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699512880640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761732.381171:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761732.381174:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761732.381176:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761732.381180:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1706 00000100:00080000:3.0:1789761732.381184:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761732.381188:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761732.381193:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761732.381208:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512880768/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761732.381218:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761732.381231:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761732.381235:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761732.381238:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761732.381243:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1706 00000100:00080000:3.0:1789761732.381248:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761732.381253:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761732.381258:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761732/real 1789761732] req@ffff8dc0107adc00 x1876699512880512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789761732.381259:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761732.381278:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512880896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761732.381280:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699512880512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761732.381309:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761732.381313:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761732.381317:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512880512:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761732.381336:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512880640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761732.381352:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761732.381361:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761732/real 1789761732] req@ffff8dc0107afb80 x1876699512880640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761732.381369:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699512880640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761732.381375:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761732.381377:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761732.381379:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512880640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761732.381383:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512880768:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761732.381397:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761732.381408:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761732/real 1789761732] req@ffff8dc0107aca80 x1876699512880768/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761732.381420:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512880768/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761732.381429:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761732.381432:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761732.381436:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512880768:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761732.381443:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512880896:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761732.381464:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761732.381473:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761732/real 1789761732] req@ffff8dc0107ae300 x1876699512880896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761732.381481:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512880896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761732.381486:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761732.381488:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761732.381490:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512880896:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761733.405048:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761734.429092:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761735.453092:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761736.477084:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761737.501074:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761737.501087:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761737.501092:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761737.501107:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1712 00000100:00080000:3.0:1789761737.501121:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761737.501138:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789761737.501179:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761737.501190:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512881024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761737.501220:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761737.501225:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761737.501227:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761737.501232:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1712 00000100:00080000:3.0:1789761737.501237:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761737.501244:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761737.501256:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761737.501260:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512881024:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761737.501276:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512881152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761737.501296:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761737.501300:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761737.501303:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761737.501313:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1712 00000100:00080000:3.0:1789761737.501319:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761737.501323:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761737.501330:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761737.501345:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512881280/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761737.501352:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761737.501366:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761737.501370:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761737.501373:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761737.501378:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1712 00000100:00080000:3.0:1789761737.501383:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761737.501388:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761737.501404:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761737/real 1789761737] req@ffff8dc0107ac700 x1876699512881024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761737.501426:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512881024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761737.501436:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761737.501439:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761737.501442:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512881024:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761737.501466:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512881152:192.168.203.156@tcp:38:kworker.0 00000008:00100000:3.0:1789761737.501485:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761737.501500:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761737.501504:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512881408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761737.501515:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761737/real 1789761737] req@ffff8dc0107ac000 x1876699512881152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761737.501528:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512881152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761737.501535:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761737.501538:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761737.501540:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512881152:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761737.501546:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512881280:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761737.501560:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761737.501567:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761737/real 1789761737] req@ffff8dc0107ad500 x1876699512881280/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761737.501574:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512881280/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761737.501580:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761737.501582:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761737.501584:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512881280:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761737.501593:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512881408:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761737.501604:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761737.501610:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761737/real 1789761737] req@ffff8dc0107aed80 x1876699512881408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761737.501626:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512881408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761737.501632:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761737.501634:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761737.501636:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512881408:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761738.525014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761739.549060:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761740.573017:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761741.597055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761742.621045:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761742.621053:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761742.621059:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761742.621068:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1717 00000100:00080000:1.0:1789761742.621076:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761742.621082:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761742.621085:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761742.621113:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512881536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761742.621135:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761742.621138:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761742.621139:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761742.621142:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1717 00000100:00080000:1.0:1789761742.621145:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761742.621147:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761742.621156:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761742.621167:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512881664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761742.621181:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512881536:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761742.621185:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761742.621192:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761742.621194:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761742.621197:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1717 00000100:00080000:1.0:1789761742.621200:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761742.621202:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761742.621207:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761742.621217:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512881792/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761742.621226:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761742.621229:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761742.621230:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761742.621234:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1717 00000100:00080000:1.0:1789761742.621250:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761742.621253:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761742.621256:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761742.621268:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512881920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761742.621269:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761742.621320:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761742/real 1789761742] req@ffff8dc005c97800 x1876699512881536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761742.621345:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512881536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761742.621355:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761742.621358:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761742.621362:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512881536:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761742.621387:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512881664:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761742.621403:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761742.621413:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761742/real 1789761742] req@ffff8dc005c94a80 x1876699512881664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761742.621434:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512881664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761742.621444:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761742.621448:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761742.621452:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512881664:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761742.621459:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512881792:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761742.621482:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761742.621492:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761742/real 1789761742] req@ffff8dc005c97100 x1876699512881792/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761742.621500:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512881792/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761742.621506:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761742.621509:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761742.621511:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512881792:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761742.621515:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512881920:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761742.621528:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761742.621535:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761742/real 1789761742] req@ffff8dc005c95180 x1876699512881920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761742.621551:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512881920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761742.621561:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761742.621563:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761742.621566:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512881920:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761743.645007:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761744.669079:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761745.693056:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761746.717023:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761747.740993:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761747.741001:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761747.741005:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789761747.741013:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761747.741014:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1722 00000100:00080000:1.0:1789761747.741025:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761747.741031:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761747.741064:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512882048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761747.741085:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761747.741088:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761747.741089:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761747.741092:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1722 00000100:00080000:1.0:1789761747.741095:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761747.741098:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761747.741105:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761747.741114:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512882048:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761747.741119:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512882176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761747.741128:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761747.741130:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761747.741131:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761747.741133:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1722 00000100:00080000:1.0:1789761747.741136:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761747.741138:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761747.741141:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761747.741149:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512882304/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761747.741156:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761747.741158:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761747.741160:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761747.741163:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1722 00000100:00080000:1.0:1789761747.741165:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761747.741167:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761747.741170:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761747.741177:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512882432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761747.741211:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512882176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761747.741235:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512882304:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761747.741255:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512882432:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761748.765068:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761749.789006:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761750.813017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789761750.813360:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1789761750.813402:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789761750.813419:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761750.813461:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761750.813471:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789761750.813479:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:3.0:1789761750.813524:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761747/real 1789761750] req@ffff8dc005c95180 x1876699512882176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761750.813551:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512882176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761750.813563:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761750.813568:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761750.813573:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512882176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761750.813580:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761747/real 1789761750] req@ffff8dc005c97100 x1876699512882304/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761750.813593:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512882304/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761750.813601:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761750.813604:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761750.813607:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512882304:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761750.813614:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761747/real 1789761750] req@ffff8dc005c94a80 x1876699512882432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761750.813625:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512882432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761750.813632:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761750.813636:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761750.813639:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512882432:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761750.813646:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761747/real 1789761750] req@ffff8dc005c95c00 x1876699512882048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761750.813657:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512882048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761750.813667:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761750.813671:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761750.813674:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512882048:192.168.203.156@tcp:250:kworker.0 00010000:00010000:1.0:1789761751.837069:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761752.861026:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1789761752.861038:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761752.861039:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761752.861045:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761752.861056:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1727 00000100:00080000:3.0:1789761752.861074:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761752.861081:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761752.861116:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512882560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761752.861142:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761752.861146:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761752.861148:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761752.861152:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1727 00000100:00080000:3.0:1789761752.861156:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761752.861160:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761752.861169:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761752.861186:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512882560:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761752.861186:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512882688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761752.861210:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761752.861214:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761752.861215:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761752.861219:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1727 00000100:00080000:3.0:1789761752.861224:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761752.861228:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761752.861234:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761752.861247:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512882816/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761752.861260:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761752.861260:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761752.861265:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761752.861267:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761752.861275:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1727 00000100:00080000:3.0:1789761752.861280:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761752.861285:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761752.861291:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761752.861303:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512882944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761752.861314:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761752/real 1789761752] req@ffff8dc0107ad180 x1876699512882560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761752.861335:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512882560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761752.861347:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761752.861351:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761752.861354:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512882560:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761752.861384:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512882688:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761752.861407:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761752.861421:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761752/real 1789761752] req@ffff8dc0107af480 x1876699512882688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761752.861434:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512882688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761752.861442:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761752.861446:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761752.861450:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512882688:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761752.861456:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512882816:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761752.861476:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761752.861488:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761752/real 1789761752] req@ffff8dc0107adf80 x1876699512882816/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761752.861500:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512882816/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761752.861514:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761752.861517:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761752.861521:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512882816:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761752.861528:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512882944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761752.861547:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761752.861559:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761752/real 1789761752] req@ffff8dc0107ac380 x1876699512882944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761752.861571:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512882944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761752.861578:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761752.861581:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761752.861584:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512882944:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761753.885005:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761754.909047:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761755.932990:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761756.957017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761757.981017:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761757.981027:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761757.981033:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761757.981044:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1732 00010000:00010000:2.0:1789761757.981048:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761757.981054:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761757.981063:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761757.981113:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512883072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761757.981146:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761757.981150:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761757.981152:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761757.981157:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1732 00000100:00080000:1.0:1789761757.981162:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761757.981167:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761757.981178:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761757.981191:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512883200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761757.981196:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512883072:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761757.981203:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761757.981207:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761757.981208:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761757.981212:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1732 00000100:00080000:1.0:1789761757.981215:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761757.981217:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761757.981221:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761757.981230:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512883328/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761757.981238:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761757.981241:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761757.981242:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761757.981245:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1732 00000100:00080000:1.0:1789761757.981256:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761757.981259:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761757.981262:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761757.981271:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512883456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761757.981273:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761757.981320:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761757/real 1789761757] req@ffff8dc005c97800 x1876699512883072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761757.981341:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512883072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761757.981360:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761757.981365:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761757.981370:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512883072:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761757.981405:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512883200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761757.981424:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761757.981435:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761757/real 1789761757] req@ffff8dc005c94700 x1876699512883200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761757.981443:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512883200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761757.981449:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761757.981452:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761757.981454:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512883200:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761757.981458:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512883328:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761757.981471:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761757.981477:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761757/real 1789761757] req@ffff8dc005c95f80 x1876699512883328/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761757.981485:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512883328/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761757.981491:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761757.981493:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761757.981495:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512883328:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761757.981499:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512883456:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761757.981510:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761757.981515:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761757/real 1789761757] req@ffff8dc005c95500 x1876699512883456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761757.981523:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512883456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761757.981528:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761757.981530:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761757.981531:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512883456:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761759.005075:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761760.029021:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761761.053113:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761762.077022:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761763.101017:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761763.101025:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761763.101028:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1789761763.101034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761763.101038:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1737 00000100:00080000:3.0:1789761763.101049:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761763.101055:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761763.101087:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512883584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761763.101109:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761763.101112:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761763.101113:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761763.101116:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1737 00000100:00080000:3.0:1789761763.101118:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761763.101121:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761763.101128:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761763.101137:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512883712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761763.101146:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512883584:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761763.101147:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761763.101155:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761763.101159:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761763.101162:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1737 00000100:00080000:3.0:1789761763.101164:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761763.101166:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761763.101169:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761763.101178:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512883840/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761763.101185:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761763.101188:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761763.101189:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761763.101192:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1737 00000100:00080000:3.0:1789761763.101194:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761763.101196:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761763.101200:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761763.101206:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512883968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761763.101221:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761763.101265:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761763/real 1789761763] req@ffff8dc0107af800 x1876699512883584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761763.101287:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512883584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761763.101300:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761763.101304:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761763.101309:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512883584:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761763.101331:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512883712:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761763.101352:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761763.101361:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761763/real 1789761763] req@ffff8dc0107ae680 x1876699512883712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761763.101371:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512883712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761763.101376:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761763.101379:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761763.101381:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512883712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761763.101385:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512883840:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761763.101397:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761763.101403:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761763/real 1789761763] req@ffff8dc0107af100 x1876699512883840/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761763.101411:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512883840/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761763.101416:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761763.101418:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761763.101420:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512883840:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761763.101423:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512883968:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761763.101433:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761763.101440:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761763/real 1789761763] req@ffff8dc0107ad880 x1876699512883968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761763.101459:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512883968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761763.101467:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761763.101470:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761763.101474:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512883968:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761764.125056:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761765.149013:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761766.173064:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761767.197054:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761768.221021:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761768.221021:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761768.221031:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761768.221035:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761768.221044:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1742 00000100:00080000:3.0:1789761768.221053:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761768.221059:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761768.221091:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512884096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761768.221111:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761768.221114:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761768.221116:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761768.221118:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1742 00000100:00080000:3.0:1789761768.221121:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761768.221124:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761768.221131:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761768.221142:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512884224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761768.221146:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512884096:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761768.221156:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761768.221159:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761768.221160:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761768.221164:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1742 00000100:00080000:3.0:1789761768.221166:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761768.221168:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761768.221171:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761768.221183:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512884352/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761768.221190:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761768.221192:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761768.221193:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761768.221196:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1742 00000100:00080000:3.0:1789761768.221201:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761768.221203:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761768.221206:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761768.221214:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761768.221215:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512884480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761768.221273:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761768/real 1789761768] req@ffff8dc0107aea00 x1876699512884096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761768.221292:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512884096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761768.221300:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761768.221303:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761768.221306:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512884096:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761768.221333:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512884224:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761768.221352:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761768.221362:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761768/real 1789761768] req@ffff8dc0107ace00 x1876699512884224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761768.221370:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512884224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761768.221377:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761768.221380:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761768.221384:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512884224:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761768.221391:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512884352:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761768.221412:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761768.221425:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761768/real 1789761768] req@ffff8dc0107ad880 x1876699512884352/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761768.221437:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512884352/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761768.221443:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761768.221445:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761768.221448:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512884352:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761768.221452:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512884480:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761768.221466:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761768.221474:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761768/real 1789761768] req@ffff8dc0107af100 x1876699512884480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761768.221481:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512884480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761768.221486:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761768.221488:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761768.221491:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512884480:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761769.245110:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761770.269048:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761771.293041:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761772.317020:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761773.341032:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761773.341046:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761773.341052:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761773.341067:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1747 00000100:00080000:1.0:1789761773.341091:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761773.341099:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789761773.341115:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761773.341146:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512884608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761773.341173:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761773.341177:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761773.341179:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761773.341183:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1747 00000100:00080000:1.0:1789761773.341188:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761773.341193:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761773.341207:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761773.341225:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512884608:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761773.341227:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512884736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761773.341249:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761773.341252:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761773.341254:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761773.341259:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1747 00000100:00080000:1.0:1789761773.341262:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761773.341265:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761773.341271:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761773.341286:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512884864/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761773.341298:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761773.341303:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761773.341308:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761773.341309:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761773.341325:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1747 00000100:00080000:1.0:1789761773.341330:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761773.341336:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761773.341341:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761773.341355:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512884992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761773.341378:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761773/real 1789761773] req@ffff8dc005c96a00 x1876699512884608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761773.341406:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512884608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761773.341418:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761773.341423:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761773.341427:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512884608:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761773.341454:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512884736:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761773.341491:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761773.341504:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761773/real 1789761773] req@ffff8dc005c94380 x1876699512884736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761773.341518:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512884736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761773.341527:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761773.341531:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761773.341536:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512884736:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761773.341544:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512884864:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761773.341570:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761773.341586:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761773/real 1789761773] req@ffff8dc005c97b80 x1876699512884864/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761773.341599:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512884864/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761773.341608:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761773.341612:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761773.341616:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512884864:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761773.341639:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512884992:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761773.341670:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761773.341688:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761773/real 1789761773] req@ffff8dc005c96300 x1876699512884992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761773.341703:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512884992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761773.341882:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761773.341887:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761773.341890:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512884992:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761774.365050:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761775.389092:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761776.413042:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761777.437024:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761778.461013:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761778.461024:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761778.461029:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761778.461040:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1753 00000100:00080000:3.0:1789761778.461049:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761778.461057:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789761778.461059:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761778.461094:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512885120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761778.461119:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761778.461122:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761778.461125:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761778.461131:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1753 00000100:00080000:3.0:1789761778.461134:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761778.461139:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761778.461150:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761778.461152:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512885120:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761778.461166:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512885248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761778.461179:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761778.461183:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761778.461185:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761778.461190:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1753 00000100:00080000:3.0:1789761778.461194:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761778.461199:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761778.461204:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761778.461209:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761778.461218:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512885376/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761778.461230:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761778.461234:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761778.461236:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761778.461240:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1753 00000100:00080000:3.0:1789761778.461244:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761778.461248:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761778.461253:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761778/real 1789761778] req@ffff8dc0107ae680 x1876699512885120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789761778.461255:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761778.461274:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512885504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761778.461275:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512885120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761778.461300:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761778.461306:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761778.461310:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512885120:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761778.461329:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512885248:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761778.461343:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761778.461351:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761778/real 1789761778] req@ffff8dc0107af800 x1876699512885248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761778.461360:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512885248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761778.461365:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761778.461367:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761778.461369:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512885248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761778.461373:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512885376:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761778.461383:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761778.461390:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761778/real 1789761778] req@ffff8dc0107ac380 x1876699512885376/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761778.461397:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512885376/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761778.461407:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761778.461409:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761778.461411:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512885376:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761778.461414:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512885504:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761778.461424:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761778.461430:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761778/real 1789761778] req@ffff8dc0107adf80 x1876699512885504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761778.461436:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512885504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761778.461441:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761778.461443:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761778.461445:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512885504:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761779.485071:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761780.509022:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761781.533030:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761782.556982:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761783.580965:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761783.580974:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761783.580978:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789761783.580984:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761783.580988:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1758 00000100:00080000:3.0:1789761783.580996:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761783.581001:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761783.581035:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512885632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761783.581052:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761783.581054:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761783.581055:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761783.581058:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1758 00000100:00080000:3.0:1789761783.581060:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761783.581063:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761783.581070:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761783.581079:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512885632:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761783.581080:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512885760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761783.581091:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761783.581093:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761783.581094:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761783.581097:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1758 00000100:00080000:3.0:1789761783.581099:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761783.581102:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761783.581106:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761783.581118:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512885888/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761783.581128:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761783.581132:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761783.581134:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761783.581137:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1758 00000800:00000100:0.0:1789761783.581142:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761783.581142:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761783.581149:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761783.581155:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761783.581166:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512886016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761783.581180:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761783/real 1789761783] req@ffff8dc0107af480 x1876699512885632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761783.581194:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512885632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761783.581208:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761783.581212:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761783.581216:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512885632:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761783.581245:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512885760:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761783.581263:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761783.581271:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761783/real 1789761783] req@ffff8dc0107ad180 x1876699512885760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761783.581278:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512885760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761783.581283:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761783.581285:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761783.581287:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512885760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761783.581290:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512885888:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761783.581301:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761783.581306:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761783/real 1789761783] req@ffff8dc0107aed80 x1876699512885888/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761783.581312:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512885888/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761783.581316:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761783.581318:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761783.581320:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512885888:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761783.581324:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512886016:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761783.581332:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761783.581337:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761783/real 1789761783] req@ffff8dc0107ad500 x1876699512886016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761783.581344:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512886016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761783.581348:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761783.581349:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761783.581351:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512886016:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761784.605062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761785.629058:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761786.653083:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761787.677061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761788.701001:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761788.701011:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761788.701016:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789761788.701023:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761788.701027:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1763 00000100:00080000:3.0:1789761788.701036:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761788.701043:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761788.701076:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512886144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761788.701100:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761788.701103:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761788.701105:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761788.701110:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1763 00000100:00080000:3.0:1789761788.701114:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761788.701118:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761788.701127:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761788.701131:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512886144:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761788.701142:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512886272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761788.701154:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761788.701157:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761788.701159:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761788.701163:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1763 00000100:00080000:3.0:1789761788.701175:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761788.701180:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789761788.701184:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789761788.701188:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761788.701203:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512886400/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761788.701217:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761788/real 1789761788] req@ffff8dc0107ac000 x1876699512886144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761788.701219:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761788.701228:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761788.701232:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761788.701237:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1763 00000100:00100000:0.0:1789761788.701241:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512886144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761788.701243:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761788.701251:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:0.0:1789761788.701254:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1789761788.701257:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789761788.701258:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761788.701264:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512886144:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761788.701274:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512886528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761788.701287:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512886272:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761788.701304:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761788.701311:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761788/real 1789761788] req@ffff8dc0107ac700 x1876699512886272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761788.701319:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512886272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761788.701324:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761788.701326:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761788.701327:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512886272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761788.701332:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512886400:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761788.701348:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761788.701357:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761788/real 1789761788] req@ffff8dc0107ae300 x1876699512886400/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761788.701368:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512886400/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761788.701376:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761788.701379:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761788.701382:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512886400:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761788.701392:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512886528:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761788.701404:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761788.701411:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761788/real 1789761788] req@ffff8dc0107aca80 x1876699512886528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761788.701422:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512886528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761788.701426:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761788.701428:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761788.701430:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512886528:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761789.724994:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761790.749059:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761791.773035:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761792.797019:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761793.821071:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761793.821083:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:1.0:1789761793.821084:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761793.821088:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761793.821109:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1768 00000100:00080000:2.0:1789761793.821123:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761793.821131:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761793.821169:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512886656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761793.821195:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761793.821199:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761793.821201:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761793.821206:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1768 00000100:00080000:2.0:1789761793.821210:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761793.821215:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761793.821225:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761793.821232:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512886656:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761793.821243:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512886784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761793.821256:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761793.821260:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761793.821262:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761793.821267:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1768 00000100:00080000:2.0:1789761793.821271:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761793.821276:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761793.821281:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761793.821295:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512886912/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761793.821297:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789761793.821310:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761793.821314:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761793.821317:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761793.821323:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1768 00000100:00080000:2.0:1789761793.821327:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761793.821332:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761793.821337:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761793.821342:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761793/real 1789761793] req@ffff8dc00368df80 x1876699512886656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761793.821353:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512887040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761793.821367:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512886656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761793.821379:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761793.821384:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761793.821389:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512886656:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761793.821411:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512886784:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761793.821433:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761793.821446:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761793/real 1789761793] req@ffff8dc00368ed80 x1876699512886784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761793.821458:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512886784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761793.821473:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761793.821477:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761793.821480:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512886784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761793.821486:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512886912:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761793.821506:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761793.821519:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761793/real 1789761793] req@ffff8dc00368e300 x1876699512886912/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761793.821530:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512886912/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761793.821538:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761793.821542:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761793.821545:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512886912:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761793.821550:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512887040:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761793.821586:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761793.821597:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761793/real 1789761793] req@ffff8dc00368f800 x1876699512887040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761793.821608:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512887040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761793.821616:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761793.821619:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761793.821623:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512887040:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761794.845002:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761795.869046:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761796.893038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761797.917021:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761798.940987:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761798.940995:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761798.940999:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761798.941010:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1773 00000100:00080000:2.0:1789761798.941020:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761798.941028:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789761798.941038:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789761798.941067:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512887168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761798.941092:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761798.941095:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761798.941097:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761798.941100:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1773 00000100:00080000:2.0:1789761798.941104:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761798.941108:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761798.941115:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761798.941121:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512887168:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761798.941128:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512887296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761798.941141:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761798.941144:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761798.941145:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761798.941148:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1773 00000100:00080000:2.0:1789761798.941150:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761798.941152:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761798.941156:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761798.941164:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512887424/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761798.941171:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761798.941173:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761798.941174:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761798.941180:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789761798.941183:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1773 00000100:00080000:2.0:1789761798.941189:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761798.941194:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761798.941199:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761798.941213:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512887552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761798.941231:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761798/real 1789761798] req@ffff8dc00368dc00 x1876699512887168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761798.941248:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512887168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761798.941257:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761798.941260:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761798.941263:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512887168:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761798.941282:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512887296:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761798.941299:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761798.941308:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761798/real 1789761798] req@ffff8dc00368e680 x1876699512887296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761798.941316:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512887296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761798.941322:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761798.941324:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761798.941326:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512887296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761798.941330:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512887424:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761798.941342:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761798.941351:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761798/real 1789761798] req@ffff8dc00368d180 x1876699512887424/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761798.941364:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512887424/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761798.941373:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761798.941377:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761798.941380:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512887424:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761798.941387:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512887552:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761798.941406:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761798.941415:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761798/real 1789761798] req@ffff8dc00368c700 x1876699512887552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761798.941423:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512887552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761798.941429:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761798.941431:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761798.941433:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512887552:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761799.965114:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761800.989042:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761802.013117:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761803.037078:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761804.061047:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761804.061055:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761804.061062:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761804.061066:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761804.061074:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1778 00000100:00080000:3.0:1789761804.061082:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761804.061087:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761804.061117:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699512887680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761804.061140:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761804.061143:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761804.061144:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761804.061147:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1778 00000100:00080000:3.0:1789761804.061149:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761804.061152:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761804.061159:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761804.061168:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699512887808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761804.061176:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761804.061178:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761804.061180:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761804.061184:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512887680:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761804.061186:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1778 00000100:00080000:3.0:1789761804.061193:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761804.061195:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761804.061199:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761804.061209:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512887936/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761804.061217:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761804.061219:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761804.061221:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761804.061223:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1778 00000100:00080000:3.0:1789761804.061226:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761804.061228:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761804.061231:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761804.061239:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512888064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761804.061246:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761804.061290:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761804/real 1789761804] req@ffff8dc0107afb80 x1876699512887680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761804.061305:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699512887680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761804.061312:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761804.061315:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761804.061319:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512887680:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761804.061336:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512887808:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761804.061351:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761804.061359:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761804/real 1789761804] req@ffff8dc0107adc00 x1876699512887808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761804.061368:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699512887808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761804.061373:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761804.061375:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761804.061378:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512887808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761804.061381:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512887936:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761804.061405:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761804.061412:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761804/real 1789761804] req@ffff8dc0107aca80 x1876699512887936/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761804.061420:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512887936/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761804.061430:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761804.061432:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761804.061434:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512887936:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761804.061438:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512888064:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761804.061448:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761804.061454:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761804/real 1789761804] req@ffff8dc0107ae300 x1876699512888064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761804.061461:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512888064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761804.061466:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761804.061468:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761804.061470:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512888064:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761805.085053:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761806.109050:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761807.133024:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761808.157055:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761809.181029:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761809.181041:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761809.181045:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761809.181054:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1783 00000100:00080000:3.0:1789761809.181063:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761809.181071:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761809.181102:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512888192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761809.181125:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761809.181128:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761809.181130:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761809.181134:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1783 00000100:00080000:3.0:1789761809.181137:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761809.181139:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761809.181147:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00010000:00010000:0.0:1789761809.181159:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761809.181159:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512888320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761809.181170:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761809.181174:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761809.181176:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761809.181182:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1783 00000100:00100000:0.0:1789761809.181185:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512888192:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761809.181186:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761809.181194:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761809.181203:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761809.181220:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512888448/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761809.181235:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761809.181241:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761809.181243:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761809.181250:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1783 00000100:00080000:3.0:1789761809.181255:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761809.181260:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789761809.181266:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789761809.181270:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761809.181288:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512888576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761809.181315:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761809/real 1789761809] req@ffff8dc0107ac700 x1876699512888192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761809.181336:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512888192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761809.181353:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761809.181358:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761809.181363:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512888192:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761809.181370:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512888320:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761809.181396:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761809.181412:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761809/real 1789761809] req@ffff8dc0107ac000 x1876699512888320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761809.181426:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512888320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761809.181434:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761809.181456:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761809.181461:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512888320:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761809.181486:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512888448:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761809.181511:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761809.181523:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761809/real 1789761809] req@ffff8dc0107ad500 x1876699512888448/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761809.181535:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512888448/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761809.181544:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761809.181547:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761809.181550:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512888448:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761809.181557:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512888576:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761809.181577:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761809.181588:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761809/real 1789761809] req@ffff8dc0107aed80 x1876699512888576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761809.181600:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512888576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761809.181608:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761809.181612:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761809.181615:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512888576:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761810.205086:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761811.229102:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761812.253058:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761813.277049:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761814.301042:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761814.301050:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761814.301053:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761814.301062:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1788 00000100:00080000:0.0:1789761814.301071:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761814.301077:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761814.301114:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149880 x1876699512888704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761814.301152:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761814.301155:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761814.301157:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761814.301160:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1788 00000100:00080000:0.0:1789761814.301164:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761814.301168:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761814.301176:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761814.301188:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b800 x1876699512888832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761814.301198:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512888704:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789761814.301201:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761814.301213:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761814.301214:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761814.301218:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1788 00000100:00080000:0.0:1789761814.301220:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761814.301223:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761814.301227:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761814.301243:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149f80 x1876699512888960/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761814.301250:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761814.301254:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761814.301255:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761814.301257:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1788 00000100:00080000:0.0:1789761814.301263:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761814.301266:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761814.301269:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761814.301276:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149180 x1876699512889088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761814.301346:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512888832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761814.301379:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512888960:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1789761814.301390:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789761814.301413:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512889088:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761815.325091:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761816.349069:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761817.373102:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789761817.373172:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1789761817.373205:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789761817.373218:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789761817.373251:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789761817.373263:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789761817.373276:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1789761817.373305:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761814/real 1789761817] req@ffff8dc00514b800 x1876699512888832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761817.373343:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b800 x1876699512888832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761817.373358:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761817.373363:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761817.373368:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512888832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761817.373396:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761814/real 1789761817] req@ffff8dc005149f80 x1876699512888960/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761817.373412:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149f80 x1876699512888960/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761817.373421:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761817.373425:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761817.373430:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512888960:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789761817.373438:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761814/real 1789761817] req@ffff8dc005149180 x1876699512889088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761817.373463:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149180 x1876699512889088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761817.373472:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761817.373476:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761817.373480:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512889088:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761817.373489:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761814/real 1789761817] req@ffff8dc005149880 x1876699512888704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761817.373503:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149880 x1876699512888704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761817.373511:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761817.373514:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761817.373520:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512888704:192.168.203.156@tcp:250:kworker.0 00010000:00010000:0.0:1789761818.397084:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761819.421080:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761819.421091:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761819.421101:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761819.421113:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1794 00000100:00080000:2.0:1789761819.421123:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761819.421132:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789761819.421155:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789761819.421177:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512889216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761819.421207:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761819.421212:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761819.421213:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761819.421217:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1794 00000100:00080000:2.0:1789761819.421219:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761819.421222:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761819.421231:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761819.421241:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512889344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761819.421249:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761819.421252:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761819.421253:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761819.421256:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1794 00000100:00080000:2.0:1789761819.421258:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761819.421260:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761819.421263:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761819.421269:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512889216:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761819.421274:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512889472/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761819.421289:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761819.421293:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761819.421295:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761819.421299:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1794 00000100:00080000:2.0:1789761819.421303:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761819.421308:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761819.421314:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761819.421328:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512889600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761819.421389:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761819.421463:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761819/real 1789761819] req@ffff8dc00368c700 x1876699512889216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761819.421487:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512889216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761819.421501:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761819.421506:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761819.421510:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512889216:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761819.421519:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512889344:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761819.421549:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761819.421564:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761819/real 1789761819] req@ffff8dc00368d180 x1876699512889344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761819.421577:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512889344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761819.421585:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761819.421588:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761819.421591:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512889344:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761819.421635:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512889472:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761819.421659:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761819.421673:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761819/real 1789761819] req@ffff8dc00368e680 x1876699512889472/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761819.421686:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512889472/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761819.421699:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761819.421703:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761819.421707:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512889472:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761819.421713:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512889600:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761819.421736:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761819.421749:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761819/real 1789761819] req@ffff8dc00368dc00 x1876699512889600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761819.421761:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512889600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761819.421769:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761819.421773:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761819.421776:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512889600:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761820.445131:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761821.469028:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761822.493099:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761823.517143:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761824.541042:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1789761824.541052:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761824.541053:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761824.541059:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761824.541070:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1799 00000100:00080000:1.0:1789761824.541082:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761824.541090:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761824.541132:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512889728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761824.541159:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761824.541163:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761824.541165:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761824.541170:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1799 00000100:00080000:1.0:1789761824.541174:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761824.541178:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761824.541188:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761824.541192:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512889728:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761824.541205:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512889856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761824.541218:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761824.541222:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761824.541225:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761824.541230:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1799 00000100:00080000:1.0:1789761824.541235:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761824.541239:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761824.541245:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761824.541248:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761824.541260:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512889984/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761824.541271:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761824.541275:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761824.541277:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761824.541281:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1799 00000100:00080000:1.0:1789761824.541285:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761824.541287:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761824/real 1789761824] req@ffff8dc005c94e00 x1876699512889728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761824.541292:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761824.541300:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761824.541320:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512889728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761824.541320:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512890112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761824.541361:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761824.541368:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761824.541372:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512889728:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761824.541392:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512889856:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761824.541411:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761824.541421:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761824/real 1789761824] req@ffff8dc005c97480 x1876699512889856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761824.541430:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512889856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761824.541435:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761824.541438:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761824.541440:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512889856:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761824.541443:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512889984:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761824.541455:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761824.541462:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761824/real 1789761824] req@ffff8dc005c96680 x1876699512889984/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761824.541469:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512889984/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761824.541474:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761824.541476:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761824.541478:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512889984:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761824.541482:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512890112:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761824.541492:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761824.541498:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761824/real 1789761824] req@ffff8dc005c95880 x1876699512890112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761824.541505:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512890112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761824.541510:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761824.541512:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761824.541514:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512890112:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761825.565176:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761826.589107:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761827.613032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761828.637044:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761829.661024:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761829.661033:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761829.661036:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1789761829.661045:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761829.661045:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1804 00000100:00080000:3.0:1789761829.661055:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761829.661060:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761829.661091:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512890240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761829.661125:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761829.661128:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761829.661129:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761829.661132:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1804 00000100:00080000:3.0:1789761829.661135:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761829.661138:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761829.661145:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761829.661155:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512890368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761829.661159:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512890240:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761829.661168:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761829.661171:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761829.661173:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761829.661176:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1804 00000100:00080000:3.0:1789761829.661180:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761829.661182:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761829.661185:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761829.661194:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512890496/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761829.661202:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761829.661204:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761829.661205:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761829.661208:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1804 00000100:00080000:3.0:1789761829.661211:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761829.661214:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761829.661217:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761829.661219:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761829.661228:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512890624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761829.661256:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761829/real 1789761829] req@ffff8dc0107ad180 x1876699512890240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761829.661269:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512890240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761829.661277:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761829.661279:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761829.661282:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512890240:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761829.661317:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512890368:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761829.661350:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761829.661361:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761829/real 1789761829] req@ffff8dc0107af480 x1876699512890368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761829.661370:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512890368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761829.661376:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761829.661378:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761829.661380:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512890368:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761829.661385:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512890496:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761829.661398:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761829.661405:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761829/real 1789761829] req@ffff8dc0107adf80 x1876699512890496/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761829.661412:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512890496/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761829.661424:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761829.661427:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761829.661429:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512890496:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761829.661433:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512890624:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761829.661444:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761829.661451:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761829/real 1789761829] req@ffff8dc0107ac380 x1876699512890624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761829.661458:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512890624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761829.661464:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761829.661466:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761829.661468:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512890624:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761830.685174:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761831.709021:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761832.733031:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761833.757027:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761834.781055:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1789761834.781059:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761834.781064:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761834.781069:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761834.781077:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1809 00000100:00080000:1.0:1789761834.781086:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761834.781091:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761834.781124:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96d80 x1876699512890752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761834.781145:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761834.781148:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761834.781149:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761834.781151:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1809 00000100:00080000:1.0:1789761834.781154:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761834.781156:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761834.781162:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761834.781173:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512890880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761834.781180:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761834.781183:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761834.781184:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761834.781189:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1809 00000100:00100000:0.0:1789761834.781191:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512890752:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761834.781191:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761834.781196:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761834.781201:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761834.781208:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512891008/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761834.781216:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761834.781218:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761834.781219:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761834.781222:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1809 00000100:00080000:1.0:1789761834.781225:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761834.781227:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761834.781230:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761834.781237:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512891136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761834.781277:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761834.781320:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761834/real 1789761834] req@ffff8dc005c96d80 x1876699512890752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761834.781340:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96d80 x1876699512890752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761834.781351:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761834.781356:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761834.781360:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512890752:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761834.781394:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512890880:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761834.781420:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761834.781434:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761834/real 1789761834] req@ffff8dc005c95880 x1876699512890880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761834.781448:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512890880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761834.781457:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761834.781461:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761834.781465:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512890880:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761834.781471:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512891008:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761834.781493:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761834.781504:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761834/real 1789761834] req@ffff8dc005c96680 x1876699512891008/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761834.781512:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512891008/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761834.781518:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761834.781520:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761834.781523:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512891008:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761834.781527:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512891136:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761834.781539:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761834.781545:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761834/real 1789761834] req@ffff8dc005c97480 x1876699512891136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761834.781552:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512891136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761834.781557:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761834.781559:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761834.781561:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512891136:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761835.805082:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761836.829024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761837.853053:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761838.877090:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761839.900984:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1789761839.900987:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761839.900994:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761839.900998:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761839.901009:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1814 00000100:00080000:1.0:1789761839.901018:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761839.901025:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761839.901060:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512891264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761839.901086:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761839.901090:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761839.901091:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761839.901096:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1814 00000100:00080000:1.0:1789761839.901099:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761839.901105:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761839.901114:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512891264:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1789761839.901115:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761839.901136:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512891392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761839.901149:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761839.901153:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761839.901155:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761839.901160:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1814 00000100:00080000:1.0:1789761839.901166:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761839.901171:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789761839.901174:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1789761839.901179:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761839.901196:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512891520/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761839.901207:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761839.901228:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1789761839.901229:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761839/real 1789761839] req@ffff8dc005c94e00 x1876699512891264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761839.901231:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761839.901242:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1814 00000100:00080000:1.0:1789761839.901248:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761839.901254:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761839.901261:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761839.901265:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512891264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761839.901276:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1789761839.901277:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512891648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761839.901281:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761839.901290:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512891264:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761839.901313:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512891392:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761839.901333:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761839.901342:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761839/real 1789761839] req@ffff8dc005c96300 x1876699512891392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761839.901350:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512891392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761839.901356:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761839.901358:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761839.901360:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512891392:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761839.901364:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512891520:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761839.901376:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761839.901382:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761839/real 1789761839] req@ffff8dc005c97b80 x1876699512891520/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761839.901390:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512891520/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761839.901395:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761839.901397:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761839.901399:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512891520:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761839.901402:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512891648:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761839.901412:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761839.901418:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761839/real 1789761839] req@ffff8dc005c94380 x1876699512891648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761839.901426:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512891648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761839.901431:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761839.901433:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761839.901435:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512891648:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761840.925074:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761841.949030:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761842.973074:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761843.997057:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761845.021052:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761845.021061:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761845.021066:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761845.021077:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1819 00000100:00080000:2.0:1789761845.021086:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761845.021094:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789761845.021114:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789761845.021133:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512891776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761845.021160:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761845.021165:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761845.021167:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761845.021190:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1819 00000100:00080000:2.0:1789761845.021195:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761845.021201:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761845.021211:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761845.021216:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512891776:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761845.021232:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512891904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761845.021247:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761845.021251:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761845.021253:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761845.021258:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1819 00000100:00080000:2.0:1789761845.021262:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761845.021267:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761845.021273:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761845.021279:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761845.021290:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512892032/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761845.021302:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761845.021306:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761845.021308:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761845.021313:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1819 00000100:00080000:2.0:1789761845.021317:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761845.021322:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761845/real 1789761845] req@ffff8dc00368f800 x1876699512891776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761845.021325:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761845.021334:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761845.021351:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512892160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761845.021355:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512891776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761845.021380:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761845.021385:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761845.021391:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512891776:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761845.021415:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512891904:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761845.021440:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761845.021454:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761845/real 1789761845] req@ffff8dc00368e300 x1876699512891904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761845.021468:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512891904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761845.021477:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761845.021480:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761845.021484:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512891904:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761845.021494:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512892032:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761845.021516:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761845.021527:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761845/real 1789761845] req@ffff8dc00368ed80 x1876699512892032/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761845.021540:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512892032/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761845.021549:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761845.021552:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761845.021555:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512892032:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761845.021562:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512892160:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761845.021591:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761845.021607:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761845/real 1789761845] req@ffff8dc00368df80 x1876699512892160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761845.021622:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512892160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761845.021633:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761845.021638:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761845.021644:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512892160:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761846.045117:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761847.069121:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761848.093042:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761849.117054:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761850.141002:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761850.141012:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761850.141017:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761850.141028:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1824 00000100:00080000:2.0:1789761850.141037:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761850.141044:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761850.141105:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512892288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1789761850.141112:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761850.141129:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761850.141192:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761850.141195:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761850.141201:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1824 00000100:00080000:2.0:1789761850.141206:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761850.141212:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761850.141218:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512892288:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789761850.141225:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761850.141245:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512892416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761850.141265:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761850.141268:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761850.141269:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761850.141272:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1824 00000100:00080000:2.0:1789761850.141276:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761850.141279:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761850.141283:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761850.141292:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512892544/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761850.141300:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789761850.141303:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789761850.141304:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761850.141308:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761850.141312:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1824 00000100:00080000:2.0:1789761850.141316:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761850.141319:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761850.141323:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761850.141331:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512892672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761850.141357:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761850/real 1789761850] req@ffff8dc00368ca80 x1876699512892288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761850.141380:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512892288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761850.141393:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761850.141398:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761850.141402:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512892288:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761850.141428:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512892416:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761850.141457:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761850.141473:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761850/real 1789761850] req@ffff8dc00368c000 x1876699512892416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761850.141488:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512892416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761850.141509:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761850.141513:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761850.141517:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512892416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761850.141524:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512892544:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761850.141548:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761850.141562:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761850/real 1789761850] req@ffff8dc00368ea00 x1876699512892544/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761850.141577:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512892544/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761850.141587:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761850.141591:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761850.141594:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512892544:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761850.141601:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512892672:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761850.141624:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761850.141646:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761850/real 1789761850] req@ffff8dc00368ce00 x1876699512892672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761850.141661:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512892672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761850.141671:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761850.141675:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761850.141678:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512892672:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761851.165086:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761852.189075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761853.213168:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761854.237062:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761855.261031:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761855.261044:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761855.261050:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761855.261063:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1829 00000100:00080000:3.0:1789761855.261077:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761855.261086:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789761855.261104:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761855.261144:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512892800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761855.261176:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761855.261181:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761855.261183:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761855.261188:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1829 00000100:00080000:3.0:1789761855.261192:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761855.261197:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761855.261208:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761855.261212:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512892800:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761855.261224:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512892928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761855.261241:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761855.261245:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761855.261246:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761855.261251:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1829 00000100:00080000:3.0:1789761855.261254:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761855.261257:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761855.261262:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761855.261273:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512893056/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761855.261283:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761855.261286:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761855.261288:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789761855.261290:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761855.261292:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1829 00000100:00080000:3.0:1789761855.261295:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761855.261298:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761855.261303:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761855.261311:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512893184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761855.261341:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761855/real 1789761855] req@ffff8dc0107af800 x1876699512892800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761855.261361:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512892800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761855.261373:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761855.261376:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761855.261380:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512892800:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761855.261404:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512892928:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761855.261423:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761855.261433:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761855/real 1789761855] req@ffff8dc0107ae680 x1876699512892928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761855.261441:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512892928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761855.261447:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761855.261450:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761855.261452:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512892928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761855.261457:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512893056:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761855.261470:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761855.261476:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761855/real 1789761855] req@ffff8dc0107af100 x1876699512893056/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761855.261484:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512893056/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761855.261490:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761855.261492:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761855.261493:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512893056:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761855.261497:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512893184:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761855.261508:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761855.261514:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761855/real 1789761855] req@ffff8dc0107ad880 x1876699512893184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761855.261521:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512893184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761855.261526:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761855.261528:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761855.261530:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512893184:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761856.285066:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761857.309073:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761858.333106:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761859.357161:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761860.381085:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761860.381117:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761860.381124:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761860.381138:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1834 00010000:00010000:0.0:1789761860.381144:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761860.381153:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761860.381165:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761860.381216:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512893312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761860.381249:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761860.381254:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761860.381257:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761860.381263:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1834 00000100:00080000:2.0:1789761860.381268:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761860.381273:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761860.381286:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761860.381287:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512893312:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761860.381313:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512893440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761860.381331:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761860.381336:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761860.381338:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761860.381343:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1834 00000100:00080000:2.0:1789761860.381351:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761860.381359:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789761860.381359:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761860.381373:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761860.381396:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512893568/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761860.381411:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789761860.381412:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761860/real 1789761860] req@ffff8dc00368f480 x1876699512893312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761860.381438:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761860.381441:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761860.381445:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512893312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761860.381452:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1834 00000100:00080000:0.0:1789761860.381464:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761860.381465:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761860.381479:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1789761860.381483:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761860.381488:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512893312:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789761860.381497:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761860.381513:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512893440:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761860.381527:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512893696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761860.381542:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761860.381579:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761860/real 1789761860] req@ffff8dc00368fb80 x1876699512893440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761860.381589:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512893440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761860.381595:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761860.381598:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761860.381600:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512893440:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761860.381605:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512893568:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761860.381620:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761860.381627:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761860/real 1789761860] req@ffff8dc00368d880 x1876699512893568/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761860.381634:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512893568/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761860.381651:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761860.381653:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761860.381656:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512893568:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761860.381666:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512893696:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761860.381678:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761860.381691:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761860/real 1789761860] req@ffff8dc00368f100 x1876699512893696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761860.381700:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512893696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761860.381705:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761860.381707:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761860.381709:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512893696:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761861.405038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761862.429073:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761863.453118:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761864.477117:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761865.501122:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:1.0:1789761865.501133:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761865.501135:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761865.501143:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761865.501164:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1840 00000100:00080000:2.0:1789761865.501179:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761865.501191:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761865.501272:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512893824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761865.501310:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761865.501315:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761865.501318:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761865.501323:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1840 00000100:00080000:2.0:1789761865.501347:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761865.501351:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761865.501359:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512893824:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789761865.501364:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761865.501382:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512893952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761865.501411:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761865.501416:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761865.501419:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761865.501428:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1840 00000100:00080000:2.0:1789761865.501434:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761865.501439:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761865.501446:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789761865.501486:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761865.501488:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512894080/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761865.501506:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761865.501512:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761865.501516:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761865.501527:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1840 00000100:00080000:2.0:1789761865.501535:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761865.501544:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761865.501548:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761865/real 1789761865] req@ffff8dc00368f100 x1876699512893824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789761865.501562:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761865.501573:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512893824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761865.501586:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761865.501593:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761865.501609:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512893824:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761865.501612:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512894208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761865.501637:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512893952:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761865.501661:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761865.501670:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761865/real 1789761865] req@ffff8dc00368d880 x1876699512893952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761865.501678:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512893952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761865.501684:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761865.501687:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761865.501689:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512893952:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761865.501693:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512894080:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761865.501705:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761865.501711:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761865/real 1789761865] req@ffff8dc00368fb80 x1876699512894080/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761865.501718:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512894080/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761865.501723:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761865.501725:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761865.501727:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512894080:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761865.501731:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512894208:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761865.501743:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761865.501749:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761865/real 1789761865] req@ffff8dc00368f480 x1876699512894208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761865.501757:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512894208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761865.501762:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761865.501764:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761865.501765:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512894208:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761866.525071:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761867.549110:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761868.573101:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761869.597063:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761870.621094:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761870.621113:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761870.621118:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761870.621131:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1845 00000100:00080000:0.0:1789761870.621141:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761870.621149:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761870.621197:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149c00 x1876699512894336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761870.621229:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761870.621232:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761870.621233:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761870.621236:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1845 00000100:00080000:0.0:1789761870.621239:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761870.621241:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761870.621252:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761870.621263:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b480 x1876699512894464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761870.621271:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761870.621274:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761870.621276:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761870.621278:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1845 00000100:00080000:0.0:1789761870.621280:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761870.621295:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761870.621298:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512894336:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1789761870.621301:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761870.621318:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514bb80 x1876699512894592/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761870.621329:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761870.621333:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761870.621334:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761870.621339:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1845 00000100:00080000:0.0:1789761870.621343:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761870.621346:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761870.621350:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761870.621361:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148380 x1876699512894720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761870.621392:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761870.621457:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761870/real 1789761870] req@ffff8dc005149c00 x1876699512894336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761870.621480:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149c00 x1876699512894336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1789761870.621490:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761870.621494:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761870.621500:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761870.621505:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512894336:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761870.621514:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512894464:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761870.621540:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761870.621558:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761870/real 1789761870] req@ffff8dc00514b480 x1876699512894464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761870.621577:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b480 x1876699512894464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761870.621590:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761870.621595:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761870.621602:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512894464:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761870.621634:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512894592:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789761870.621660:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761870.621678:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761870/real 1789761870] req@ffff8dc00514bb80 x1876699512894592/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761870.621696:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514bb80 x1876699512894592/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761870.621709:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761870.621715:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761870.621721:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512894592:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789761870.621730:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512894720:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761870.621757:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761870.621769:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761870/real 1789761870] req@ffff8dc005148380 x1876699512894720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761870.621781:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148380 x1876699512894720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761870.621789:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761870.621792:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761870.621795:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512894720:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761871.645146:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761872.669062:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761873.693142:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761874.717097:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761875.741047:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761875.741056:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761875.741060:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761875.741068:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1850 00000100:00080000:0.0:1789761875.741076:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761875.741082:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761875.741114:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148000 x1876699512894848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761875.741135:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761875.741138:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761875.741139:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761875.741142:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1850 00000100:00080000:0.0:1789761875.741144:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761875.741147:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761875.741154:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761875.741163:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514ad80 x1876699512894976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761875.741172:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789761875.741172:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512894848:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789761875.741179:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761875.741182:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761875.741186:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1850 00000100:00080000:0.0:1789761875.741188:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761875.741190:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761875.741194:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761875.741202:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148700 x1876699512895104/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761875.741214:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761875.741217:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761875.741218:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761875.741220:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1850 00000100:00080000:0.0:1789761875.741223:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761875.741225:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761875.741228:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789761875.741234:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761875.741238:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514aa00 x1876699512895232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1789761875.741277:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789761875.741286:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761875/real 1789761875] req@ffff8dc005148000 x1876699512894848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761875.741302:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148000 x1876699512894848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761875.741311:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761875.741314:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761875.741317:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512894848:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761875.741335:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512894976:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761875.741349:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761875.741358:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761875/real 1789761875] req@ffff8dc00514ad80 x1876699512894976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761875.741366:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514ad80 x1876699512894976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761875.741372:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761875.741374:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761875.741376:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512894976:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761875.741380:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512895104:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789761875.741390:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761875.741396:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761875/real 1789761875] req@ffff8dc005148700 x1876699512895104/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761875.741404:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148700 x1876699512895104/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761875.741413:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761875.741415:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761875.741417:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512895104:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789761875.741420:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512895232:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761875.741438:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761875.741444:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761875/real 1789761875] req@ffff8dc00514aa00 x1876699512895232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761875.741451:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514aa00 x1876699512895232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761875.741457:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761875.741459:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761875.741461:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512895232:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761876.765104:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761877.789075:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761878.813020:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761879.837083:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761880.861024:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761880.861031:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761880.861035:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761880.861044:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1855 00010000:00010000:1.0:1789761880.861045:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761880.861052:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761880.861059:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761880.861089:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512895360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761880.861111:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761880.861114:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761880.861116:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761880.861120:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1855 00000100:00080000:3.0:1789761880.861125:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761880.861128:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761880.861136:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761880.861142:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512895360:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761880.861150:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512895488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761880.861159:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761880.861162:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761880.861163:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761880.861166:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1855 00000100:00080000:3.0:1789761880.861170:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761880.861173:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761880.861178:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761880.861189:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512895616/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761880.861201:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761880.861204:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1789761880.861226:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512895488:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761880.861252:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512895616:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1789761880.861265:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761880.861271:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1855 00000100:00080000:3.0:1789761880.861275:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761880.861278:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761880.861284:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761880.861296:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512895744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761880.861329:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512895744:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761881.885127:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761882.909087:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789761883.933346:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00010000:00010000:2.0:1789761883.933402:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789761883.933421:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789761883.933441:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789761883.933482:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789761883.933495:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789761883.933506:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1789761883.933520:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761880/real 1789761883] req@ffff8dc0107aea00 x1876699512895488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761883.933553:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512895488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761883.933568:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761883.933573:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761883.933577:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512895488:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761883.933587:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761880/real 1789761883] req@ffff8dc0107ad880 x1876699512895616/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761883.933597:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512895616/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761883.933603:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761883.933606:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761883.933609:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512895616:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789761883.933613:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761880/real 1789761883] req@ffff8dc0107ace00 x1876699512895360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761883.933624:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512895360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761883.933633:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761883.933636:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761883.933640:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512895360:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761883.933671:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761880/real 1789761883] req@ffff8dc0107af100 x1876699512895744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761883.933686:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512895744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761883.933695:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761883.933700:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761883.933703:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512895744:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761884.957099:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761885.981039:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761885.981051:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761885.981055:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761885.981067:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1860 00000100:00080000:2.0:1789761885.981077:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761885.981084:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789761885.981129:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512895872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1789761885.981146:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761885.981169:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761885.981175:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761885.981177:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761885.981181:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1860 00000100:00080000:2.0:1789761885.981186:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761885.981192:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761885.981205:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761885.981223:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512896000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761885.981231:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512895872:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789761885.981236:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761885.981243:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761885.981245:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761885.981250:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1860 00000100:00080000:2.0:1789761885.981255:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761885.981260:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761885.981267:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761885.981282:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512896128/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761885.981295:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761885.981299:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761885.981301:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761885.981306:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1860 00000100:00080000:2.0:1789761885.981322:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789761885.981327:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789761885.981331:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761885.981339:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761885.981353:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512896256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761885.981404:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761885/real 1789761885] req@ffff8dc00368ce00 x1876699512895872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761885.981426:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512895872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761885.981438:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761885.981442:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761885.981446:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512895872:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761885.981477:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512896000:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761885.981512:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761885.981527:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761885/real 1789761885] req@ffff8dc00368ea00 x1876699512896000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761885.981540:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512896000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761885.981549:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761885.981553:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761885.981557:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512896000:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761885.981564:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512896128:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761885.981589:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761885.981602:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761885/real 1789761885] req@ffff8dc00368c000 x1876699512896128/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761885.981614:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512896128/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761885.981623:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761885.981627:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761885.981631:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512896128:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761885.981637:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512896256:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761885.981660:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761885.981673:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761885/real 1789761885] req@ffff8dc00368ca80 x1876699512896256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761885.981684:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512896256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761885.981693:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761885.981697:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761885.981701:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512896256:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761887.005158:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761888.029089:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761889.053067:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761890.077044:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761891.101024:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761891.101035:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761891.101040:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761891.101050:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1865 00000100:00080000:3.0:1789761891.101062:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1789761891.101067:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761891.101071:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761891.101112:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512896384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761891.101140:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761891.101145:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761891.101147:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761891.101152:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1865 00000100:00080000:3.0:1789761891.101156:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761891.101161:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761891.101171:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761891.101178:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512896384:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761891.101189:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512896512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761891.101210:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761891.101215:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761891.101217:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761891.101222:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1865 00000100:00080000:3.0:1789761891.101233:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761891.101237:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1789761891.101241:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789761891.101246:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761891.101261:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512896640/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761891.101272:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761891.101276:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761891.101278:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761891.101282:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1865 00000100:00080000:3.0:1789761891.101286:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789761891.101287:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761891/real 1789761891] req@ffff8dc0107ae680 x1876699512896384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761891.101296:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761891.101304:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761891.101308:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512896384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761891.101318:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789761891.101321:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512896768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761891.101324:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761891.101331:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512896384:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761891.101356:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512896512:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761891.101373:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761891.101382:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761891/real 1789761891] req@ffff8dc0107af800 x1876699512896512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761891.101390:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512896512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761891.101396:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761891.101398:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761891.101400:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512896512:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761891.101404:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512896640:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789761891.101414:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761891.101420:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761891/real 1789761891] req@ffff8dc0107ac380 x1876699512896640/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761891.101427:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512896640/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761891.101432:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761891.101434:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761891.101436:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512896640:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789761891.101439:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512896768:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761891.101449:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761891.101455:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761891/real 1789761891] req@ffff8dc0107adf80 x1876699512896768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761891.101467:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512896768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761891.101472:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761891.101473:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761891.101475:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512896768:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761892.125046:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761893.149040:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761894.173095:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761895.197121:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761896.220967:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761896.220974:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761896.220978:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761896.220986:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1870 00000100:00080000:3.0:1789761896.220994:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761896.221000:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761896.221015:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761896.221032:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512896896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761896.221056:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761896.221060:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761896.221061:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761896.221065:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1870 00000100:00080000:3.0:1789761896.221069:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761896.221072:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761896.221079:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761896.221082:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512896896:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761896.221091:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512897024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761896.221099:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761896.221102:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761896.221103:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761896.221106:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1870 00000100:00080000:3.0:1789761896.221109:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761896.221112:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761896.221115:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761896.221123:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512897152/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761896.221130:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761896.221132:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761896.221134:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761896.221137:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1870 00000800:00000100:2.0:1789761896.221142:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761896.221143:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761896.221148:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761896.221152:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761896.221161:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512897280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761896.221182:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761896/real 1789761896] req@ffff8dc0107af480 x1876699512896896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761896.221196:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512896896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761896.221204:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761896.221207:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761896.221210:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512896896:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761896.221226:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512897024:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761896.221240:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761896.221248:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761896/real 1789761896] req@ffff8dc0107ad180 x1876699512897024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761896.221255:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512897024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761896.221261:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761896.221263:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761896.221265:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512897024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761896.221269:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512897152:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789761896.221279:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761896.221285:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761896/real 1789761896] req@ffff8dc0107aed80 x1876699512897152/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761896.221292:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512897152/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761896.221297:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761896.221299:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761896.221301:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512897152:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789761896.221305:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512897280:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761896.221314:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761896.221320:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761896/real 1789761896] req@ffff8dc0107ad500 x1876699512897280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761896.221327:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512897280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761896.221332:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761896.221333:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761896.221336:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512897280:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761897.245059:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761898.269048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761899.293018:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761900.317031:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761901.340979:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761901.340987:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761901.340990:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761901.340999:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1875 00000100:00080000:3.0:1789761901.341007:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761901.341012:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761901.341045:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512897408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1789761901.341046:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761901.341068:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761901.341071:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761901.341072:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761901.341075:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1875 00000100:00080000:3.0:1789761901.341078:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761901.341081:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761901.341089:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761901.341098:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512897536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761901.341102:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512897408:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761901.341119:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761901.341121:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761901.341122:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761901.341126:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1875 00000100:00080000:3.0:1789761901.341128:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761901.341130:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761901.341133:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761901.341141:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512897664/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761901.341148:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761901.341150:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761901.341152:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761901.341155:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1875 00000100:00080000:3.0:1789761901.341157:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761901.341160:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761901.341162:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789761901.341164:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761901.341172:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512897792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789761901.341202:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761901/real 1789761901] req@ffff8dc0107ac000 x1876699512897408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761901.341217:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512897408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761901.341225:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761901.341228:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761901.341230:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512897408:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761901.341247:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512897536:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761901.341262:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761901.341270:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761901/real 1789761901] req@ffff8dc0107ac700 x1876699512897536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761901.341278:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512897536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761901.341284:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761901.341286:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761901.341288:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512897536:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761901.341292:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512897664:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789761901.341314:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761901.341321:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761901/real 1789761901] req@ffff8dc0107ae300 x1876699512897664/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761901.341329:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512897664/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761901.341335:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761901.341337:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761901.341339:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512897664:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789761901.341343:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512897792:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761901.341353:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761901.341359:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761901/real 1789761901] req@ffff8dc0107aca80 x1876699512897792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761901.341366:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512897792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761901.341372:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761901.341374:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761901.341376:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512897792:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761902.364994:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761903.389041:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761904.413073:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761905.437052:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761906.461046:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761906.461058:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761906.461063:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789761906.461069:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761906.461076:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1881 00000100:00080000:3.0:1789761906.461088:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761906.461097:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761906.461140:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699512897920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761906.461167:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761906.461170:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761906.461172:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761906.461175:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1881 00000100:00080000:3.0:1789761906.461177:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761906.461180:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761906.461189:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761906.461200:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699512898048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761906.461209:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761906.461211:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761906.461213:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789761906.461221:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512897920:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761906.461221:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1881 00000100:00080000:3.0:1789761906.461226:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761906.461234:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761906.461244:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761906.461257:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512898176/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761906.461265:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761906.461269:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761906.461270:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761906.461273:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1881 00000100:00080000:3.0:1789761906.461275:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761906.461279:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761906.461283:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761906.461296:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512898304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789761906.461323:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761906.461382:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761906/real 1789761906] req@ffff8dc0107adc00 x1876699512897920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761906.461411:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699512897920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761906.461423:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761906.461428:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761906.461433:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512897920:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761906.461459:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512898048:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761906.461497:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761906.461513:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761906/real 1789761906] req@ffff8dc0107afb80 x1876699512898048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761906.461528:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699512898048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761906.461537:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761906.461541:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761906.461545:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512898048:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789761906.461552:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512898176:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789761906.461574:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761906.461587:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761906/real 1789761906] req@ffff8dc0107aca80 x1876699512898176/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761906.461601:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512898176/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761906.461617:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761906.461620:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761906.461629:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512898176:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789761906.461637:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512898304:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789761906.461660:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761906.461673:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761906/real 1789761906] req@ffff8dc0107ae300 x1876699512898304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789761906.461685:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512898304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789761906.461694:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789761906.461698:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789761906.461702:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512898304:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761907.485122:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761908.509101:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761909.533094:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761910.557096:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761911.580998:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761911.581006:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761911.581010:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761911.581018:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1886 00000100:00080000:2.0:1789761911.581026:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761911.581031:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789761911.581042:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789761911.581065:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512898432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761911.581083:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761911.581086:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761911.581088:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761911.581091:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1886 00000100:00080000:2.0:1789761911.581094:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761911.581097:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761911.581105:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761911.581114:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512898432:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789761911.581115:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512898560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761911.581129:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761911.581131:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761911.581132:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761911.581134:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1886 00000100:00080000:2.0:1789761911.581137:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761911.581139:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761911.581142:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761911.581151:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512898688/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761911.581158:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761911.581161:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761911.581162:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761911.581164:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1886 00000100:00080000:2.0:1789761911.581174:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761911.581176:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761911.581180:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789761911.581183:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789761911.581190:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512898816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761911.581228:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761911/real 1789761911] req@ffff8dc00368df80 x1876699512898432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761911.581251:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512898432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761911.581259:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761911.581262:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761911.581265:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512898432:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761911.581283:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512898560:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761911.581299:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761911.581308:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761911/real 1789761911] req@ffff8dc00368ed80 x1876699512898560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761911.581326:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512898560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761911.581335:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761911.581339:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761911.581343:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512898560:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761911.581349:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512898688:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761911.581370:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761911.581379:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761911/real 1789761911] req@ffff8dc00368e300 x1876699512898688/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761911.581388:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512898688/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761911.581393:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761911.581395:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761911.581397:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512898688:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761911.581401:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512898816:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761911.581415:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761911.581421:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761911/real 1789761911] req@ffff8dc00368f800 x1876699512898816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761911.581428:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512898816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761911.581432:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761911.581434:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761911.581437:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512898816:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761912.605075:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761913.629112:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761914.653037:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761915.676991:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761916.701010:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761916.701017:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761916.701021:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789761916.701024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761916.701030:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1891 00000100:00080000:3.0:1789761916.701039:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761916.701044:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761916.701073:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512898944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761916.701096:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761916.701099:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761916.701101:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761916.701104:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1891 00000100:00080000:3.0:1789761916.701106:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761916.701110:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761916.701117:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761916.701127:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512899072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761916.701135:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512898944:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761916.701138:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761916.701145:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761916.701147:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761916.701151:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1891 00000100:00080000:3.0:1789761916.701154:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761916.701158:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761916.701162:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761916.701172:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512899200/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761916.701179:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761916.701182:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761916.701183:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761916.701186:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1891 00000100:00080000:3.0:1789761916.701189:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761916.701191:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761916.701195:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761916.701203:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512899328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761916.701208:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761916.701258:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761916/real 1789761916] req@ffff8dc0107ac700 x1876699512898944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761916.701277:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512898944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761916.701288:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761916.701293:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761916.701298:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512898944:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761916.701327:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512899072:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761916.701351:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761916.701366:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761916/real 1789761916] req@ffff8dc0107ac000 x1876699512899072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761916.701380:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512899072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761916.701399:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761916.701402:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761916.701406:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512899072:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761916.701413:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512899200:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761916.701431:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761916.701440:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761916/real 1789761916] req@ffff8dc0107ad500 x1876699512899200/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761916.701447:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512899200/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761916.701453:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761916.701455:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761916.701457:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512899200:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761916.701461:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512899328:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761916.701472:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761916.701488:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761916/real 1789761916] req@ffff8dc0107aed80 x1876699512899328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761916.701495:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512899328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761916.701500:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761916.701502:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761916.701504:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512899328:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761917.725027:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761918.749071:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761919.773022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761920.797063:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761921.821003:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761921.821019:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761921.821028:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761921.821033:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761921.821044:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1896 00000100:00080000:3.0:1789761921.821056:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761921.821063:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761921.821106:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512899456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761921.821133:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761921.821136:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761921.821138:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761921.821142:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1896 00000100:00080000:3.0:1789761921.821145:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761921.821148:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761921.821158:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761921.821170:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512899584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761921.821178:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512899456:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761921.821183:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761921.821191:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761921.821193:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761921.821204:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1896 00000100:00080000:3.0:1789761921.821206:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761921.821210:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761921.821215:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761921.821225:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512899712/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761921.821233:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761921.821237:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761921.821239:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761921.821242:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1896 00000100:00080000:3.0:1789761921.821246:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789761921.821252:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761921.821252:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761921.821263:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761921.821274:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512899840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761921.821301:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761921/real 1789761921] req@ffff8dc0107ad180 x1876699512899456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761921.821316:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512899456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761921.821325:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761921.821328:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761921.821331:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512899456:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761921.821352:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512899584:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761921.821367:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761921.821375:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761921/real 1789761921] req@ffff8dc0107af480 x1876699512899584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761921.821384:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512899584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761921.821394:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761921.821397:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761921.821399:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512899584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761921.821403:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512899712:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761921.821414:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761921.821420:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761921/real 1789761921] req@ffff8dc0107adf80 x1876699512899712/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761921.821428:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512899712/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761921.821433:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761921.821435:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761921.821437:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512899712:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761921.821441:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512899840:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761921.821451:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761921.821457:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761921/real 1789761921] req@ffff8dc0107ac380 x1876699512899840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761921.821464:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512899840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761921.821469:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761921.821471:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761921.821473:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512899840:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761922.845032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761923.869075:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761924.893051:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761925.917036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761926.940973:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761926.940981:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761926.940985:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761926.940994:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1901 00000100:00080000:0.0:1789761926.941003:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761926.941009:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761926.941046:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148a80 x1876699512899968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761926.941068:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761926.941071:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761926.941072:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761926.941075:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1901 00000100:00080000:0.0:1789761926.941078:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761926.941080:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761926.941087:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761926.941097:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514aa00 x1876699512900096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761926.941105:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761926.941107:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761926.941108:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761926.941111:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1901 00000100:00080000:0.0:1789761926.941113:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761926.941118:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761926.941120:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512899968:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1789761926.941123:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761926.941133:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148700 x1876699512900224/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761926.941141:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761926.941144:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761926.941145:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761926.941148:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1901 00000100:00080000:0.0:1789761926.941151:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761926.941153:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761926.941156:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761926.941163:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514ad80 x1876699512900352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789761926.941181:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00010000:00010000:0.0:1789761926.941201:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761926.941227:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761926/real 1789761926] req@ffff8dc005148a80 x1876699512899968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761926.941243:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148a80 x1876699512899968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761926.941252:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761926.941256:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761926.941258:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512899968:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761926.941264:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512900096:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761926.941280:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761926.941288:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761926/real 1789761926] req@ffff8dc00514aa00 x1876699512900096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761926.941296:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514aa00 x1876699512900096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761926.941302:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761926.941304:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761926.941306:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512900096:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761926.941324:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512900224:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761926.941348:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761926.941361:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761926/real 1789761926] req@ffff8dc005148700 x1876699512900224/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761926.941375:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148700 x1876699512900224/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761926.941385:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761926.941389:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761926.941393:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512900224:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761926.941399:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512900352:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761926.941416:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761926.941424:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761926/real 1789761926] req@ffff8dc00514ad80 x1876699512900352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761926.941432:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514ad80 x1876699512900352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761926.941438:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761926.941440:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761926.941442:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512900352:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761927.965008:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761928.989090:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761930.013058:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761931.037254:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761932.061044:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761932.061055:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761932.061060:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761932.061072:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1906 00000100:00080000:0.0:1789761932.061082:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761932.061091:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761932.061146:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148000 x1876699512900480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761932.061174:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761932.061177:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761932.061179:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761932.061182:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1906 00000100:00080000:0.0:1789761932.061186:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761932.061189:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761932.061197:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761932.061207:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148380 x1876699512900608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761932.061216:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761932.061219:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789761932.061220:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761932.061225:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1906 00000100:00100000:1.0:1789761932.061227:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512900480:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789761932.061228:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761932.061236:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761932.061240:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761932.061249:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514bb80 x1876699512900736/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761932.061257:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761932.061260:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761932.061262:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761932.061264:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1906 00000100:00080000:0.0:1789761932.061278:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761932.061283:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761932.061286:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761932.061293:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761932.061297:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b480 x1876699512900864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761932.061334:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761932/real 1789761932] req@ffff8dc005148000 x1876699512900480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761932.061350:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148000 x1876699512900480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1789761932.061356:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761932.061360:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761932.061366:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761932.061369:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512900480:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761932.061391:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512900608:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761932.061411:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761932.061424:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761932/real 1789761932] req@ffff8dc005148380 x1876699512900608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761932.061433:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148380 x1876699512900608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761932.061439:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761932.061441:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761932.061443:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512900608:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761932.061448:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512900736:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761932.061460:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761932.061467:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761932/real 1789761932] req@ffff8dc00514bb80 x1876699512900736/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761932.061475:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514bb80 x1876699512900736/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761932.061481:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761932.061483:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761932.061485:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512900736:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761932.061489:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512900864:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761932.061500:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761932.061507:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761932/real 1789761932] req@ffff8dc00514b480 x1876699512900864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761932.061515:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b480 x1876699512900864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761932.061537:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761932.061540:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761932.061542:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512900864:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761933.085048:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761934.109120:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761935.133052:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761936.157092:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761937.181008:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761937.181020:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:0.0:1789761937.181022:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761937.181024:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761937.181038:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1911 00000100:00080000:3.0:1789761937.181048:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761937.181056:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761937.181107:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512900992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761937.181137:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761937.181140:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761937.181142:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761937.181146:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1911 00000100:00080000:3.0:1789761937.181150:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761937.181154:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761937.181164:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761937.181167:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512900992:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761937.181179:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512901120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761937.181188:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761937.181191:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761937.181193:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761937.181196:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1911 00000100:00080000:3.0:1789761937.181198:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761937.181202:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761937.181206:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761937.181215:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512901248/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761937.181223:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761937.181226:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761937.181228:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761937.181231:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1911 00000100:00080000:3.0:1789761937.181233:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761937.181235:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761937.181238:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761937.181245:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761937.181249:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512901376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761937.181293:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761937/real 1789761937] req@ffff8dc0107af800 x1876699512900992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761937.181310:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512900992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761937.181319:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761937.181322:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761937.181325:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512900992:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761937.181347:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512901120:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761937.181361:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761937.181369:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761937/real 1789761937] req@ffff8dc0107ae680 x1876699512901120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761937.181378:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512901120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761937.181384:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761937.181386:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761937.181388:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512901120:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761937.181392:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512901248:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761937.181403:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761937.181408:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761937/real 1789761937] req@ffff8dc0107af100 x1876699512901248/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761937.181416:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512901248/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761937.181421:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761937.181423:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761937.181425:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512901248:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761937.181428:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512901376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761937.181437:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761937.181443:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761937/real 1789761937] req@ffff8dc0107ace00 x1876699512901376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761937.181451:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512901376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761937.181460:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761937.181462:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761937.181464:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512901376:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761938.205109:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761939.229053:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761940.253081:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761941.277062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789761942.301032:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761942.301043:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789761942.301047:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761942.301058:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1916 00000100:00080000:0.0:1789761942.301068:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761942.301076:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761942.301123:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149c00 x1876699512901504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761942.301151:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761942.301155:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761942.301157:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761942.301161:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1916 00000100:00080000:0.0:1789761942.301166:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761942.301170:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761942.301180:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761942.301195:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149880 x1876699512901632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761942.301207:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761942.301212:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789761942.301214:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512901504:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789761942.301215:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761942.301224:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1916 00000100:00080000:0.0:1789761942.301229:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761942.301232:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761942.301238:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789761942.301250:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149180 x1876699512901760/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761942.301262:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789761942.301265:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789761942.301268:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789761942.301272:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1916 00000100:00080000:0.0:1789761942.301276:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789761942.301280:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789761942.301286:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789761942.301286:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761942.301303:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149f80 x1876699512901888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761942.301339:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761942/real 1789761942] req@ffff8dc005149c00 x1876699512901504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761942.301363:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149c00 x1876699512901504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761942.301375:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761942.301380:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761942.301384:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512901504:192.168.203.156@tcp:250:kworker.0 00010000:00010000:0.0:1789761942.301399:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761942.301414:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512901632:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761942.301456:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761942.301472:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761942/real 1789761942] req@ffff8dc005149880 x1876699512901632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761942.301486:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149880 x1876699512901632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761942.301495:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761942.301499:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761942.301503:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512901632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761942.301510:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512901760:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789761942.301530:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761942.301541:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761942/real 1789761942] req@ffff8dc005149180 x1876699512901760/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761942.301566:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149180 x1876699512901760/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761942.301575:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761942.301578:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761942.301582:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512901760:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761942.301589:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512901888:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789761942.301609:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761942.301621:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761942/real 1789761942] req@ffff8dc005149f80 x1876699512901888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789761942.301633:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149f80 x1876699512901888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789761942.301641:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789761942.301644:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789761942.301648:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512901888:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761943.325046:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761944.349049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761945.373093:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761946.397062:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761947.421049:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761947.421062:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761947.421068:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789761947.421080:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761947.421082:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1922 00000100:00080000:3.0:1789761947.421097:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761947.421105:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761947.421158:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512902016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761947.421184:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761947.421187:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761947.421188:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761947.421191:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1922 00000100:00080000:3.0:1789761947.421194:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761947.421197:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761947.421206:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761947.421215:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512902144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761947.421223:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761947.421226:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761947.421227:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761947.421230:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1922 00000100:00080000:3.0:1789761947.421232:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761947.421235:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761947.421239:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761947.421245:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512902016:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761947.421252:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512902272/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761947.421261:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761947.421263:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761947.421265:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761947.421272:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1922 00000100:00080000:3.0:1789761947.421275:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761947.421278:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761947.421282:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761947.421291:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512902400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789761947.421353:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512902144:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789761947.421388:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512902272:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789761947.421418:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512902400:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761948.445076:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761949.469134:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761950.493221:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789761950.493221:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:1.0:1789761950.493287:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789761950.493304:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789761950.493367:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789761950.493393:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789761950.493407:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:3.0:1789761950.493513:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761947/real 1789761950] req@ffff8dc0107ace00 x1876699512902272/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761950.493555:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512902272/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761950.493574:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761950.493581:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761950.493587:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512902272:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761950.493598:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761947/real 1789761950] req@ffff8dc0107af100 x1876699512902400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761950.493615:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512902400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761950.493625:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761950.493629:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761950.493633:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512902400:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761950.493642:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761947/real 1789761950] req@ffff8dc0107ad880 x1876699512902016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761950.493665:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512902016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761950.493673:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761950.493678:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761950.493682:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512902016:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761950.493691:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761947/real 1789761950] req@ffff8dc0107aea00 x1876699512902144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761950.493705:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512902144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761950.493715:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761950.493719:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761950.493723:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512902144:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761951.517107:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761952.541014:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761952.541031:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761952.541037:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761952.541051:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1927 00000100:00080000:3.0:1789761952.541063:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761952.541071:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761952.541107:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512902528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1789761952.541108:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761952.541132:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761952.541140:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761952.541143:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761952.541149:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1927 00000100:00080000:3.0:1789761952.541154:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789761952.541159:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512902528:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761952.541165:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761952.541178:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761952.541197:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512902656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761952.541210:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761952.541214:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761952.541216:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761952.541222:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1927 00000800:00000100:0.0:1789761952.541228:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761952.541229:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761952.541237:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761952.541244:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761952.541258:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512902784/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761952.541269:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761952.541274:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761952.541276:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789761952.541281:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761952/real 1789761952] req@ffff8dc0107aea00 x1876699512902528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761952.541282:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1927 00000100:00080000:3.0:1789761952.541288:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761952.541297:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761952.541304:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761952.541309:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512902528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761952.541321:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512902912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789761952.541322:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761952.541332:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761952.541341:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512902528:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761952.541386:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512902656:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761952.541415:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761952.541432:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761952/real 1789761952] req@ffff8dc0107ad880 x1876699512902656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761952.541445:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512902656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761952.541461:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761952.541465:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761952.541469:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512902656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761952.541476:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512902784:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761952.541498:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761952.541509:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761952/real 1789761952] req@ffff8dc0107af100 x1876699512902784/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761952.541521:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512902784/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761952.541529:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761952.541532:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761952.541536:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512902784:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761952.541542:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512902912:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761952.541561:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761952.541573:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761952/real 1789761952] req@ffff8dc0107ace00 x1876699512902912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761952.541587:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512902912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761952.541595:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761952.541597:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761952.541601:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512902912:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761953.565165:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761954.589012:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761955.613096:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761956.637039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761957.661046:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761957.661055:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761957.661059:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761957.661069:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1932 00000100:00080000:3.0:1789761957.661078:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1789761957.661085:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761957.661085:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761957.661123:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512903040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761957.661144:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761957.661147:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761957.661148:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761957.661151:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1932 00000100:00080000:3.0:1789761957.661154:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761957.661157:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761957.661164:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761957.661173:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512903168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761957.661178:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512903040:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789761957.661186:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761957.661189:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761957.661190:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761957.661201:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1932 00000100:00080000:3.0:1789761957.661204:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761957.661206:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761957.661210:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761957.661220:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512903296/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761957.661227:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761957.661230:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761957.661231:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761957.661233:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1932 00000100:00080000:3.0:1789761957.661237:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789761957.661241:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761957.661244:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761957.661248:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761957.661257:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512903424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761957.661279:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761957/real 1789761957] req@ffff8dc0107ae680 x1876699512903040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761957.661294:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512903040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761957.661301:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761957.661304:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761957.661307:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512903040:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761957.661324:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512903168:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761957.661360:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761957.661370:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761957/real 1789761957] req@ffff8dc0107af800 x1876699512903168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761957.661379:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512903168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761957.661385:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761957.661387:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761957.661389:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512903168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761957.661394:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512903296:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761957.661408:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761957.661414:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761957/real 1789761957] req@ffff8dc0107ac380 x1876699512903296/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761957.661422:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512903296/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761957.661428:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761957.661430:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761957.661432:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512903296:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761957.661436:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512903424:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761957.661446:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761957.661451:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761957/real 1789761957] req@ffff8dc0107adf80 x1876699512903424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761957.661458:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512903424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761957.661464:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761957.661466:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761957.661468:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512903424:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761958.685075:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761959.709028:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761960.733019:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761961.757097:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761962.781018:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761962.781029:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761962.781033:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761962.781047:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1937 00010000:00010000:0.0:1789761962.781059:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761962.781060:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761962.781069:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761962.781116:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512903552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761962.781144:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761962.781147:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761962.781149:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761962.781152:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1937 00000100:00080000:1.0:1789761962.781155:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761962.781159:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761962.781169:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761962.781179:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512903552:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761962.781186:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512903680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761962.781194:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761962.781197:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761962.781198:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761962.781201:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1937 00000100:00080000:1.0:1789761962.781204:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761962.781206:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761962.781209:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761962.781218:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512903808/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761962.781226:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761962.781228:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761962.781230:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761962.781232:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1937 00000100:00080000:1.0:1789761962.781252:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761962.781255:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761962.781258:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761962.781266:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512903936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761962.781276:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761962.781350:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761962/real 1789761962] req@ffff8dc005c96a00 x1876699512903552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761962.781374:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512903552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761962.781388:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761962.781392:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761962.781395:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512903552:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761962.781420:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512903680:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761962.781440:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761962.781450:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761962/real 1789761962] req@ffff8dc005c95500 x1876699512903680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761962.781458:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512903680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761962.781464:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761962.781467:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761962.781469:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512903680:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761962.781472:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512903808:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761962.781484:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761962.781491:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761962/real 1789761962] req@ffff8dc005c95f80 x1876699512903808/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761962.781503:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512903808/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761962.781508:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761962.781510:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761962.781512:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512903808:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761962.781516:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512903936:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761962.781527:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761962.781533:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761962/real 1789761962] req@ffff8dc005c94700 x1876699512903936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761962.781541:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512903936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761962.781547:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761962.781550:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761962.781553:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512903936:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761963.805030:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761964.829064:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761965.853018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761966.877058:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761967.901018:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761967.901033:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761967.901040:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1789761967.901052:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761967.901055:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1942 00000100:00080000:3.0:1789761967.901082:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761967.901093:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761967.901144:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512904064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761967.901169:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761967.901176:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761967.901179:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761967.901184:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1942 00000100:00080000:3.0:1789761967.901189:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761967.901194:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761967.901201:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512904064:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789761967.901208:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761967.901227:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512904192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761967.901245:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761967.901248:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761967.901261:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761967.901264:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1942 00000100:00080000:3.0:1789761967.901268:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761967.901272:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761967.901276:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761967.901313:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512904320/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761967.901324:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761967.901324:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761967.901331:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761967.901333:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761967.901338:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1942 00000100:00080000:3.0:1789761967.901342:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761967.901346:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761967.901352:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761967.901363:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512904448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761967.901383:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761967/real 1789761967] req@ffff8dc0107af480 x1876699512904064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761967.901403:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512904064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761967.901414:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761967.901418:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761967.901422:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512904064:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761967.901447:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512904192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761967.901466:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761967.901476:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761967/real 1789761967] req@ffff8dc0107ad180 x1876699512904192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761967.901487:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512904192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761967.901494:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761967.901496:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761967.901499:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512904192:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761967.901504:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512904320:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761967.901518:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761967.901525:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761967/real 1789761967] req@ffff8dc0107aed80 x1876699512904320/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761967.901534:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512904320/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761967.901541:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761967.901543:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761967.901546:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512904320:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761967.901550:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512904448:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761967.901563:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761967.901570:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761967/real 1789761967] req@ffff8dc0107ad500 x1876699512904448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761967.901579:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512904448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761967.901585:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761967.901587:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761967.901589:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512904448:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761968.925002:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761969.949032:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761970.973060:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761971.997106:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761973.021020:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761973.021031:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761973.021036:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789761973.021038:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761973.021049:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1947 00000100:00080000:1.0:1789761973.021058:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761973.021064:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761973.021096:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512904576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761973.021116:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761973.021119:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761973.021120:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761973.021123:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1947 00000100:00080000:1.0:1789761973.021125:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761973.021128:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761973.021144:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761973.021160:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512904576:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789761973.021162:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512904704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761973.021173:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761973.021176:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761973.021177:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761973.021180:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1947 00000100:00080000:1.0:1789761973.021183:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761973.021185:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761973.021190:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761973.021208:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512904832/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761973.021233:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761973.021236:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761973.021237:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761973.021240:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1947 00000100:00080000:1.0:1789761973.021243:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761973.021248:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789761973.021249:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1789761973.021253:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761973.021266:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512904960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761973.021301:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761973/real 1789761973] req@ffff8dc005c97800 x1876699512904576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761973.021316:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512904576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789761998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761973.021325:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761973.021328:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761973.021332:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512904576:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761973.021357:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512904704:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761973.021373:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761973.021382:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761973/real 1789761973] req@ffff8dc005c95c00 x1876699512904704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761973.021391:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512904704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761973.021396:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761973.021399:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761973.021401:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512904704:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761973.021405:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512904832:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761973.021416:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761973.021422:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761973/real 1789761973] req@ffff8dc005c94a80 x1876699512904832/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761973.021430:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512904832/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789761998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761973.021435:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761973.021437:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761973.021439:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512904832:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761973.021443:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512904960:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761973.021454:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761973.021460:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761973/real 1789761973] req@ffff8dc005c97100 x1876699512904960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761973.021467:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512904960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789761998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761973.021472:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761973.021474:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761973.021476:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512904960:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789761974.045041:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761975.069073:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761976.093057:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761977.117072:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761978.140995:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761978.141006:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761978.141011:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761978.141023:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1952 00000100:00080000:1.0:1789761978.141033:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761978.141040:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789761978.141046:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789761978.141079:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512905088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761978.141104:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761978.141106:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761978.141108:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761978.141111:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1952 00000100:00080000:1.0:1789761978.141114:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761978.141117:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761978.141126:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761978.141138:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512905216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761978.141145:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512905088:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789761978.141148:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761978.141154:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761978.141156:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761978.141159:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1952 00000100:00080000:1.0:1789761978.141161:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761978.141163:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761978.141167:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761978.141188:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512905344/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761978.141196:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761978.141198:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761978.141199:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761978.141202:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1952 00000100:00080000:1.0:1789761978.141215:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761978.141219:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761978.141224:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761978.141230:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789761978.141238:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512905472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761978.141289:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761978/real 1789761978] req@ffff8dc005c95180 x1876699512905088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761978.141310:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512905088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761978.141321:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761978.141326:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761978.141330:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512905088:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761978.141354:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512905216:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761978.141381:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761978.141396:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761978/real 1789761978] req@ffff8dc005c97100 x1876699512905216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761978.141410:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512905216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761978.141426:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761978.141429:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761978.141434:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512905216:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761978.141440:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512905344:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761978.141462:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761978.141475:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761978/real 1789761978] req@ffff8dc005c94a80 x1876699512905344/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761978.141500:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512905344/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761978.141509:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761978.141512:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761978.141515:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512905344:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761978.141522:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512905472:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761978.141543:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761978.141555:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761978/real 1789761978] req@ffff8dc005c95c00 x1876699512905472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761978.141568:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512905472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761978.141577:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761978.141580:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761978.141584:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512905472:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761979.165079:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761980.189027:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761981.213089:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761982.237103:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761983.261029:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789761983.261115:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761983.261122:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789761983.261126:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761983.261149:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1957 00000100:00080000:1.0:1789761983.261160:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761983.261166:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789761983.261203:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512905600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761983.261253:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512905600:192.168.203.156@tcp:250:kworker.0 00000800:00000100:0.0:1789761983.261303:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789761983.261332:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761983.261335:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761983.261336:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761983.261339:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1957 00000100:00080000:1.0:1789761983.261342:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761983.261346:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761983.261353:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761983.261363:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512905728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761983.261421:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761983/real 1789761983] req@ffff8dc005c97800 x1876699512905600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761983.261435:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512905600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761983.261442:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761983.261445:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761983.261448:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512905600:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761983.261467:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512905728:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761983.261483:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761983.261492:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761983/real 1789761983] req@ffff8dc005c94700 x1876699512905728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761983.261500:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512905728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761983.261505:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761983.261507:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761983.261509:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512905728:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1789761983.261548:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761983.261552:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789761983.261556:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761983.261559:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1957 00000100:00080000:1.0:1789761983.261563:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761983.261565:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789761983.261571:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789761983.261581:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512905856/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789761983.261592:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789761983.261595:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789761983.261596:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789761983.261600:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1957 00000100:00080000:1.0:1789761983.261602:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789761983.261606:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789761983.261609:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512905856:192.168.203.156@tcp:8:kworker.0 00000008:00100000:1.0:1789761983.261611:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789761983.261623:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512905984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789761983.261646:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761983.261666:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761983/real 1789761983] req@ffff8dc005c95f80 x1876699512905856/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761983.261682:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512905856/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761983.261693:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761983.261697:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761983.261701:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512905856:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761983.261716:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512905984:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761983.261739:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761983.261751:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761983/real 1789761983] req@ffff8dc005c95500 x1876699512905984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761983.261764:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512905984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761983.261770:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761983.261773:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761983.261775:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512905984:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761984.285080:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761985.309048:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761986.333040:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761987.357084:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789761988.381037:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761988.381046:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789761988.381050:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761988.381061:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1962 00000100:00080000:2.0:1789761988.381070:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761988.381078:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789761988.381080:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789761988.381143:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512906112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761988.381178:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761988.381182:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761988.381184:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761988.381189:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1962 00000100:00080000:2.0:1789761988.381197:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761988.381204:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761988.381216:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761988.381238:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512906240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789761988.381243:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512906112:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789761988.381257:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761988.381261:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789761988.381263:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761988.381269:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1962 00000100:00080000:2.0:1789761988.381273:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761988.381279:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761988.381287:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789761988.381300:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512906368/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789761988.381310:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789761988.381313:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789761988.381315:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789761988.381319:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1962 00000100:00080000:2.0:1789761988.381323:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789761988.381326:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789761988.381331:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789761988.381340:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512906496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789761988.381387:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761988.381421:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761988/real 1789761988] req@ffff8dc00368dc00 x1876699512906112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761988.381435:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512906112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761988.381442:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761988.381445:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761988.381449:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512906112:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761988.381469:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512906240:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761988.381492:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761988.381506:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761988/real 1789761988] req@ffff8dc00368e680 x1876699512906240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761988.381651:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512906240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761988.381658:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761988.381661:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761988.381665:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512906240:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789761988.381671:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512906368:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789761988.381707:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761988.381767:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761988/real 1789761988] req@ffff8dc00368d180 x1876699512906368/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761988.381779:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512906368/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761988.381785:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761988.381787:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761988.381789:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512906368:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789761988.381794:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512906496:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789761988.381810:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761988.381818:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761988/real 1789761988] req@ffff8dc00368c700 x1876699512906496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789761988.381826:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512906496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789761988.381831:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789761988.381833:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789761988.381835:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512906496:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789761989.405070:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761990.429078:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761991.453090:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789761992.477086:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761993.500975:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761993.500983:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789761993.500987:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761993.500997:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1968 00000100:00080000:3.0:1789761993.501005:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761993.501011:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789761993.501026:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789761993.501045:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512906624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761993.501080:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761993.501083:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761993.501085:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761993.501088:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1968 00000100:00080000:3.0:1789761993.501090:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761993.501094:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761993.501102:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761993.501115:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512906624:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761993.501116:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512906752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761993.501129:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761993.501132:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761993.501133:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761993.501136:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1968 00000100:00080000:3.0:1789761993.501138:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761993.501141:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761993.501144:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761993.501154:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512906880/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761993.501161:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761993.501163:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761993.501165:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761993.501167:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1968 00000100:00080000:3.0:1789761993.501169:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761993.501172:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761993.501174:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789761993.501179:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789761993.501183:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512907008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761993.501217:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761993/real 1789761993] req@ffff8dc0107ac000 x1876699512906624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761993.501231:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512906624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761993.501239:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761993.501242:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761993.501245:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512906624:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761993.501262:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512906752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761993.501276:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761993.501284:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761993/real 1789761993] req@ffff8dc0107ac700 x1876699512906752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761993.501292:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512906752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761993.501297:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761993.501299:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761993.501302:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512906752:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761993.501305:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512906880:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761993.501315:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761993.501321:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761993/real 1789761993] req@ffff8dc0107ae300 x1876699512906880/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761993.501329:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512906880/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761993.501334:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761993.501336:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761993.501338:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512906880:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761993.501342:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512907008:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761993.501353:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761993.501358:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761993/real 1789761993] req@ffff8dc0107aca80 x1876699512907008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761993.501366:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512907008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761993.501370:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761993.501372:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761993.501374:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512907008:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789761994.525137:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789761995.549135:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789761996.573087:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789761997.597073:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761998.621062:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761998.621073:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1789761998.621076:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789761998.621078:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761998.621091:0:8:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1973 00000100:00080000:3.0:1789761998.621102:0:8:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761998.621110:0:8:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789761998.621148:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699512907136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761998.621175:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761998.621179:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761998.621182:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761998.621187:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1973 00000100:00080000:3.0:1789761998.621191:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761998.621196:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761998.621205:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789761998.621218:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512907136:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789761998.621224:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699512907264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761998.621239:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789761998.621243:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789761998.621245:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761998.621250:0:8:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1973 00000100:00080000:3.0:1789761998.621254:0:8:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761998.621259:0:8:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761998.621266:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789761998.621285:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512907392/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789761998.621297:0:8:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789761998.621300:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789761998.621303:0:8:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789761998.621308:0:8:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789761998.621314:0:8:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1973 00000100:00080000:3.0:1789761998.621320:0:8:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789761998.621325:0:8:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789761998.621331:0:8:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789761998.621347:0:8:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512907520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789761998.621361:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761998/real 1789761998] req@ffff8dc0107afb80 x1876699512907136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761998.621393:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699512907136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761998.621404:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761998.621408:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761998.621412:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512907136:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789761998.621444:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512907264:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761998.621471:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761998.621483:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761998/real 1789761998] req@ffff8dc0107adc00 x1876699512907264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761998.621492:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699512907264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761998.621498:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761998.621501:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761998.621503:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512907264:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789761998.621507:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512907392:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789761998.621522:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761998.621533:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761998/real 1789761998] req@ffff8dc0107aca80 x1876699512907392/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761998.621554:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512907392/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761998.621564:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761998.621567:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761998.621571:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512907392:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789761998.621577:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512907520:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789761998.621595:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789761998.621604:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789761998/real 1789761998] req@ffff8dc0107ae300 x1876699512907520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789761998.621616:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512907520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789761998.621625:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789761998.621629:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789761998.621632:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512907520:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789761999.645047:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762000.669096:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762001.693119:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762002.717061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762003.741067:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762003.741078:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762003.741083:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762003.741095:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1978 00000100:00080000:0.0:1789762003.741107:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762003.741114:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762003.741157:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b800 x1876699512907648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762003.741185:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762003.741189:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762003.741190:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762003.741193:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1978 00000100:00080000:0.0:1789762003.741196:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762003.741199:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762003.741208:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762003.741219:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514a300 x1876699512907776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762003.741227:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512907648:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789762003.741229:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762003.741236:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762003.741237:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762003.741241:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1978 00000100:00080000:0.0:1789762003.741242:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762003.741245:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762003.741248:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762003.741258:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148e00 x1876699512907904/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762003.741266:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762003.741268:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762003.741269:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762003.741272:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1978 00000100:00080000:0.0:1789762003.741275:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762003.741277:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762003.741280:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762003.741287:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b100 x1876699512908032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762003.741313:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762003.741359:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762003/real 1789762003] req@ffff8dc00514b800 x1876699512907648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762003.741374:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b800 x1876699512907648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762003.741387:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762003.741392:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762003.741396:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512907648:192.168.203.156@tcp:250:kworker.0 00010000:00010000:1.0:1789762003.741402:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762003.741422:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512907776:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762003.741439:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762003.741448:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762003/real 1789762003] req@ffff8dc00514a300 x1876699512907776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762003.741456:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514a300 x1876699512907776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762003.741462:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762003.741464:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762003.741466:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512907776:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762003.741470:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512907904:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789762003.741482:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762003.741488:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762003/real 1789762003] req@ffff8dc005148e00 x1876699512907904/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762003.741507:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148e00 x1876699512907904/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762003.741516:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762003.741519:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762003.741523:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512907904:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789762003.741530:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512908032:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762003.741551:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762003.741560:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762003/real 1789762003] req@ffff8dc00514b100 x1876699512908032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762003.741568:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b100 x1876699512908032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762003.741573:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762003.741576:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762003.741578:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512908032:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762004.765094:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762005.789043:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762006.813016:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762007.837105:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762008.861013:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762008.861021:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762008.861024:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762008.861034:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1983 00000100:00080000:0.0:1789762008.861043:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762008.861049:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762008.861083:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149500 x1876699512908160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762008.861102:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762008.861104:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762008.861105:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762008.861108:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1983 00000100:00080000:0.0:1789762008.861110:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762008.861113:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762008.861120:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762008.861130:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b100 x1876699512908288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762008.861140:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789762008.861140:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512908160:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789762008.861145:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762008.861148:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762008.861153:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1983 00000100:00080000:0.0:1789762008.861156:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762008.861158:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762008.861161:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762008.861169:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148e00 x1876699512908416/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762008.861176:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762008.861179:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762008.861180:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762008.861182:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1983 00000100:00080000:0.0:1789762008.861185:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762008.861188:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762008.861191:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762008.861197:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514a300 x1876699512908544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762008.861215:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762008.861259:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762008/real 1789762008] req@ffff8dc005149500 x1876699512908160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762008.861281:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149500 x1876699512908160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:2.0:1789762008.861293:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762008.861305:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762008.861314:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762008.861320:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512908160:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762008.861356:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512908288:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762008.861380:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762008.861395:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762008/real 1789762008] req@ffff8dc00514b100 x1876699512908288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762008.861409:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b100 x1876699512908288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762008.861418:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762008.861422:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762008.861427:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512908288:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762008.861434:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512908416:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789762008.861455:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762008.861465:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762008/real 1789762008] req@ffff8dc005148e00 x1876699512908416/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762008.861478:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148e00 x1876699512908416/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762008.861487:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762008.861490:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762008.861494:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512908416:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789762008.861500:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512908544:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762008.861521:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762008.861532:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762008/real 1789762008] req@ffff8dc00514a300 x1876699512908544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762008.861545:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514a300 x1876699512908544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762008.861553:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762008.861557:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762008.861560:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512908544:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762009.885096:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762010.909044:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762011.933105:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762012.957103:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762013.981115:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762013.981197:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762013.981206:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762013.981210:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762013.981222:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1988 00000100:00080000:2.0:1789762013.981234:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762013.981241:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762013.981288:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512908672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762013.981315:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762013.981319:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762013.981320:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762013.981324:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1988 00000100:00080000:2.0:1789762013.981326:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762013.981329:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762013.981338:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762013.981349:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512908800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762013.981358:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762013.981363:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1789762013.981364:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512908672:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762013.981365:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762013.981373:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1988 00000100:00080000:2.0:1789762013.981376:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762013.981378:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762013.981381:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762013.981395:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512908928/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762013.981407:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762013.981411:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762013.981413:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762013.981418:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1988 00000100:00080000:2.0:1789762013.981438:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762013.981443:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762013.981449:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762013.981461:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512909056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762013.981485:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512908800:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762013.981517:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512908928:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789762013.981542:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512909056:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762015.005122:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762016.029087:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762017.053167:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762017.053206:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:1.0:1789762017.053255:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762017.053272:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762017.053322:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762017.053333:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762017.053342:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:3.0:1789762017.053421:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762013/real 1789762017] req@ffff8dc00368d180 x1876699512908800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762017.053447:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512908800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762017.053460:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762017.053466:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762017.053471:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512908800:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762017.053480:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762013/real 1789762017] req@ffff8dc00368e680 x1876699512908928/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762017.053494:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512908928/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762017.053502:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762017.053506:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762017.053509:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512908928:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789762017.053516:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762013/real 1789762017] req@ffff8dc00368dc00 x1876699512909056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762017.053528:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512909056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762017.053536:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762017.053540:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762017.053543:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512909056:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762017.053556:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762013/real 1789762017] req@ffff8dc00368c700 x1876699512908672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762017.053570:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512908672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762017.053579:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762017.053583:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762017.053586:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512908672:192.168.203.156@tcp:250:kworker.0 00010000:00010000:2.0:1789762018.077129:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762019.101021:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762019.101029:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762019.101032:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762019.101041:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1993 00000100:00080000:1.0:1789762019.101049:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762019.101055:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762019.101086:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512909184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1789762019.101113:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762019.101118:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762019.101122:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762019.101124:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762019.101128:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1993 00000100:00080000:1.0:1789762019.101131:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762019.101135:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762019.101142:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762019.101150:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512909184:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762019.101153:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512909312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762019.101168:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762019.101171:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762019.101172:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762019.101175:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1993 00000100:00080000:1.0:1789762019.101178:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762019.101180:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762019.101185:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762019.101198:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512909440/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762019.101209:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762019.101212:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762019.101214:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789762019.101220:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789762019.101223:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1993 00000100:00080000:1.0:1789762019.101235:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762019.101241:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762019.101246:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762019.101260:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512909568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762019.101274:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762019/real 1789762019] req@ffff8dc005c96a00 x1876699512909184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762019.101295:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512909184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762019.101307:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762019.101311:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762019.101316:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512909184:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762019.101339:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512909312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762019.101365:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762019.101380:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762019/real 1789762019] req@ffff8dc005c94380 x1876699512909312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762019.101393:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512909312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762019.101403:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762019.101407:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762019.101411:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512909312:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762019.101418:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512909440:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762019.101441:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762019.101454:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762019/real 1789762019] req@ffff8dc005c97b80 x1876699512909440/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762019.101467:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512909440/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762019.101476:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762019.101480:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762019.101484:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512909440:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762019.101493:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512909568:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762019.101516:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762019.101527:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762019/real 1789762019] req@ffff8dc005c96300 x1876699512909568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762019.101539:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512909568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762019.101547:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762019.101551:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762019.101554:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512909568:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762020.125114:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762021.149057:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762022.172991:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762023.197052:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762024.221015:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762024.221024:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762024.221030:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789762024.221032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762024.221041:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 1998 00000100:00080000:2.0:1789762024.221051:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762024.221059:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762024.221094:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512909696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762024.221122:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762024.221127:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762024.221130:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762024.221136:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1998 00000100:00080000:2.0:1789762024.221141:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762024.221144:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512909696:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762024.221150:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762024.221163:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762024.221181:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512909824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762024.221195:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762024.221199:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762024.221201:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762024.221206:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 1998 00000100:00080000:2.0:1789762024.221211:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762024.221218:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762024.221220:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762024.221228:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762024.221242:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512909952/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762024.221255:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762024.221260:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1789762024.221263:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762024/real 1789762024] req@ffff8dc00368f800 x1876699512909696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762024.221263:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762024.221271:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 1998 00000100:00080000:2.0:1789762024.221277:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762024.221284:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762024.221290:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762024.221294:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512909696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762024.221309:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789762024.221314:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512910080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762024.221315:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762024.221325:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512909696:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762024.221360:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512909824:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762024.221380:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762024.221390:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762024/real 1789762024] req@ffff8dc00368e300 x1876699512909824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762024.221404:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512909824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762024.221412:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762024.221416:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762024.221419:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512909824:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762024.221426:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512909952:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762024.221447:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762024.221461:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762024/real 1789762024] req@ffff8dc00368ed80 x1876699512909952/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762024.221478:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512909952/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762024.221484:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762024.221486:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762024.221489:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512909952:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762024.221493:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512910080:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762024.221509:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762024.221517:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762024/real 1789762024] req@ffff8dc00368df80 x1876699512910080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762024.221526:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512910080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762024.221535:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762024.221539:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762024.221543:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512910080:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762025.245088:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762026.269090:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762027.293043:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762028.317006:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762029.341049:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762029.341059:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762029.341063:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762029.341073:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2003 00000100:00080000:3.0:1789762029.341100:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762029.341108:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789762029.341115:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762029.341147:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512910208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762029.341173:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762029.341176:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762029.341178:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762029.341182:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2003 00000100:00080000:3.0:1789762029.341186:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762029.341190:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762029.341199:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762029.341230:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512910208:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762029.341231:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512910336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762029.341249:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762029.341261:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762029.341263:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762029.341267:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2003 00000100:00080000:3.0:1789762029.341272:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762029.341276:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762029.341281:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762029.341293:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512910464/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762029.341304:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762029.341313:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762029.341319:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762029.341322:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762029.341330:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2003 00000100:00080000:3.0:1789762029.341336:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762029.341342:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762029.341349:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762029/real 1789762029] req@ffff8dc0107ac700 x1876699512910208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789762029.341351:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762029.341374:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512910592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762029.341381:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512910208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762029.341398:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762029.341401:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762029.341406:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512910208:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762029.341429:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512910336:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762029.341448:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762029.341458:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762029/real 1789762029] req@ffff8dc0107ac000 x1876699512910336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762029.341468:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512910336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762029.341474:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762029.341477:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762029.341480:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512910336:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762029.341484:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512910464:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762029.341497:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762029.341504:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762029/real 1789762029] req@ffff8dc0107ad500 x1876699512910464/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762029.341513:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512910464/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762029.341524:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762029.341527:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762029.341529:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512910464:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762029.341534:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512910592:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762029.341546:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762029.341553:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762029/real 1789762029] req@ffff8dc0107aed80 x1876699512910592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762029.341561:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512910592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762029.341568:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762029.341570:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762029.341572:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512910592:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762030.364993:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762031.389034:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762032.413062:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762033.437039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762034.461003:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1789762034.461008:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762034.461012:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762034.461017:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762034.461025:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2009 00000100:00080000:2.0:1789762034.461032:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762034.461038:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762034.461071:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512910720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762034.461120:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762034.461124:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762034.461125:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762034.461128:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2009 00000100:00080000:2.0:1789762034.461132:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762034.461136:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762034.461154:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762034.461178:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512910720:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762034.461183:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512910848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762034.461201:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762034.461204:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762034.461205:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762034.461208:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2009 00000100:00080000:2.0:1789762034.461210:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762034.461213:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762034.461217:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762034.461252:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512910976/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762034.461259:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762034.461262:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762034.461264:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762034.461266:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2009 00000100:00080000:2.0:1789762034.461268:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762034.461270:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789762034.461276:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:2.0:1789762034.461277:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762034.461292:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512911104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762034.461333:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762034/real 1789762034] req@ffff8dc00368ca80 x1876699512910720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762034.461350:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512910720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762034.461373:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762034.461377:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762034.461380:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512910720:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762034.461398:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512910848:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762034.461412:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762034.461421:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762034/real 1789762034] req@ffff8dc00368c000 x1876699512910848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762034.461429:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512910848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762034.461434:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762034.461436:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762034.461438:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512910848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762034.461442:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512910976:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762034.461452:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762034.461457:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762034/real 1789762034] req@ffff8dc00368ea00 x1876699512910976/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762034.461465:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512910976/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762034.461470:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762034.461472:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762034.461473:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512910976:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762034.461477:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512911104:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762034.461488:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762034.461493:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762034/real 1789762034] req@ffff8dc00368ce00 x1876699512911104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762034.461501:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512911104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762034.461506:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762034.461507:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762034.461509:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512911104:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762035.485062:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762036.509009:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762037.533129:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762038.557083:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762039.581051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762039.581051:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762039.581062:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762039.581067:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762039.581075:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2014 00000100:00080000:3.0:1789762039.581086:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762039.581092:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762039.581125:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512911232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762039.581147:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762039.581152:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762039.581153:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762039.581156:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2014 00000100:00080000:3.0:1789762039.581158:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762039.581161:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762039.581168:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762039.581177:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512911360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762039.581186:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762039.581189:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762039.581196:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762039.581200:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2014 00000100:00100000:0.0:1789762039.581202:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512911232:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762039.581202:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762039.581211:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762039.581223:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762039.581236:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512911488/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762039.581247:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762039.581251:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762039.581253:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762039.581257:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2014 00000100:00080000:3.0:1789762039.581262:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762039.581266:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762039.581273:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762039.581286:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512911616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762039.581302:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762039.581362:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762039/real 1789762039] req@ffff8dc0107ad180 x1876699512911232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762039.581385:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512911232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762039.581398:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762039.581403:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762039.581408:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512911232:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762039.581438:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512911360:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762039.581462:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762039.581475:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762039/real 1789762039] req@ffff8dc0107af480 x1876699512911360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762039.581488:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512911360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762039.581498:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762039.581502:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762039.581505:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512911360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762039.581512:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512911488:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762039.581530:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762039.581540:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762039/real 1789762039] req@ffff8dc0107adf80 x1876699512911488/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762039.581552:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512911488/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762039.581561:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762039.581564:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762039.581567:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512911488:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762039.581573:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512911616:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762039.581590:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762039.581599:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762039/real 1789762039] req@ffff8dc0107ac380 x1876699512911616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762039.581619:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512911616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762039.581628:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762039.581631:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762039.581634:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512911616:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762040.605045:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762041.629064:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762042.653075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762043.677034:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762044.700995:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762044.701002:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762044.701006:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762044.701014:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2019 00000100:00080000:3.0:1789762044.701022:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762044.701028:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789762044.701042:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762044.701061:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512911744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762044.701081:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762044.701084:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762044.701085:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762044.701088:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2019 00000100:00080000:3.0:1789762044.701090:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762044.701093:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762044.701100:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762044.701109:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512911872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762044.701116:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512911744:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762044.701119:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762044.701124:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762044.701126:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762044.701129:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2019 00000100:00080000:3.0:1789762044.701132:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762044.701134:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762044.701138:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762044.701146:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512912000/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762044.701153:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762044.701155:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762044.701157:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762044.701160:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2019 00000100:00080000:3.0:1789762044.701162:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762044.701164:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762044.701167:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762044.701174:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512912128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762044.701179:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762044.701221:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762044/real 1789762044] req@ffff8dc0107af800 x1876699512911744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762044.701236:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512911744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762044.701244:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762044.701247:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762044.701250:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512911744:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762044.701268:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512911872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762044.701282:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762044.701290:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762044/real 1789762044] req@ffff8dc0107ae680 x1876699512911872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762044.701298:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512911872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762044.701304:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762044.701306:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762044.701308:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512911872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762044.701311:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512912000:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762044.701322:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762044.701328:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762044/real 1789762044] req@ffff8dc0107ace00 x1876699512912000/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762044.701335:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512912000/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762044.701340:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762044.701342:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762044.701344:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512912000:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762044.701347:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512912128:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762044.701358:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762044.701364:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762044/real 1789762044] req@ffff8dc0107af100 x1876699512912128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762044.701371:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512912128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762044.701376:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762044.701377:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762044.701380:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512912128:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762045.725073:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762046.749027:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762047.773050:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762048.797037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762049.820996:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762049.821006:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762049.821010:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762049.821025:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2024 00010000:00010000:3.0:1789762049.821028:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762049.821034:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762049.821041:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762049.821073:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512912256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762049.821097:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762049.821100:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762049.821101:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762049.821104:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2024 00000100:00080000:1.0:1789762049.821106:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762049.821109:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762049.821121:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762049.821130:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512912256:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762049.821133:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512912384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762049.821144:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762049.821146:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762049.821148:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762049.821151:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2024 00000100:00080000:1.0:1789762049.821154:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762049.821155:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762049.821159:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762049.821168:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512912512/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762049.821175:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762049.821177:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762049.821178:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762049.821182:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2024 00000100:00080000:1.0:1789762049.821185:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762049.821188:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789762049.821193:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1789762049.821194:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762049.821205:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512912640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762049.821232:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762049/real 1789762049] req@ffff8dc005c94e00 x1876699512912256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762049.821247:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512912256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762049.821255:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762049.821258:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762049.821261:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512912256:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762049.821277:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512912384:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762049.821293:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762049.821304:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762049/real 1789762049] req@ffff8dc005c97480 x1876699512912384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762049.821317:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512912384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762049.821326:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762049.821329:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762049.821333:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512912384:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762049.821339:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512912512:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762049.821361:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762049.821370:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762049/real 1789762049] req@ffff8dc005c96680 x1876699512912512/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762049.821378:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512912512/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762049.821383:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762049.821385:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762049.821387:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512912512:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762049.821391:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512912640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762049.821403:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762049.821427:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762049/real 1789762049] req@ffff8dc005c95880 x1876699512912640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762049.821436:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512912640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762049.821442:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762049.821444:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762049.821446:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512912640:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762050.845046:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762051.869043:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762052.893019:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762053.917051:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762054.941033:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762054.941045:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762054.941049:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762054.941061:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2029 00010000:00010000:2.0:1789762054.941070:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762054.941073:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762054.941082:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762054.941126:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512912768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762054.941152:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762054.941155:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762054.941157:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762054.941160:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2029 00000100:00080000:3.0:1789762054.941163:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762054.941166:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762054.941176:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762054.941186:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512912896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762054.941195:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762054.941199:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512912768:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762054.941200:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762054.941204:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762054.941211:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2029 00000100:00080000:3.0:1789762054.941216:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762054.941219:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762054.941223:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762054.941236:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512913024/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762054.941246:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762054.941249:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762054.941251:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762054.941254:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2029 00000100:00080000:3.0:1789762054.941257:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762054.941261:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762054.941266:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762054.941275:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762054.941277:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512913152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762054.941321:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762054/real 1789762054] req@ffff8dc0107ad880 x1876699512912768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762054.941349:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512912768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762054.941361:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762054.941366:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762054.941371:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512912768:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762054.941396:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512912896:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762054.941421:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762054.941446:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762054/real 1789762054] req@ffff8dc0107aea00 x1876699512912896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762054.941460:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512912896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762054.941469:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762054.941473:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762054.941477:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512912896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762054.941484:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512913024:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762054.941505:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762054.941517:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762054/real 1789762054] req@ffff8dc0107af100 x1876699512913024/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762054.941530:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512913024/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762054.941545:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762054.941548:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762054.941551:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512913024:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762054.941558:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512913152:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762054.941581:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762054.941593:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762054/real 1789762054] req@ffff8dc0107ace00 x1876699512913152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762054.941619:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512913152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762054.941628:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762054.941631:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762054.941635:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512913152:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762055.965071:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762056.989130:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762058.013065:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762059.037181:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762060.061048:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762060.061060:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762060.061068:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1789762060.061068:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762060.061085:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2034 00000100:00080000:1.0:1789762060.061097:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762060.061105:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762060.061149:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96d80 x1876699512913280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762060.061178:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762060.061182:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762060.061184:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762060.061189:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2034 00000100:00080000:1.0:1789762060.061193:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762060.061198:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762060.061211:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762060.061225:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512913408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762060.061231:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512913280:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789762060.061246:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762060.061251:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762060.061253:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762060.061257:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2034 00000100:00080000:1.0:1789762060.061262:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762060.061265:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762060.061270:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762060.061285:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512913536/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762060.061296:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762060.061301:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762060.061303:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789762060.061312:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789762060.061317:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2034 00000100:00080000:1.0:1789762060.061326:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762060.061332:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762060.061343:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762060.061365:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762060/real 1789762060] req@ffff8dc005c96d80 x1876699512913280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762060.061366:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512913664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762060.061447:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96d80 x1876699512913280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762060.061458:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762060.061462:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762060.061466:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512913280:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762060.061496:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512913408:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762060.061532:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762060.061547:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762060/real 1789762060] req@ffff8dc005c95880 x1876699512913408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762060.061558:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512913408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762060.061581:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762060.061586:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762060.061590:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512913408:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762060.061597:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512913536:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762060.061617:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762060.061628:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762060/real 1789762060] req@ffff8dc005c96680 x1876699512913536/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762060.061637:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512913536/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762060.061649:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762060.061653:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762060.061659:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512913536:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762060.061669:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512913664:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762060.061696:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762060.061705:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762060/real 1789762060] req@ffff8dc005c97480 x1876699512913664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762060.061714:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512913664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762060.061719:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762060.061720:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762060.061722:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512913664:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762061.085048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762062.109049:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762063.133046:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762064.157049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762065.181012:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762065.181022:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762065.181026:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762065.181038:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2039 00000100:00080000:3.0:1789762065.181049:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1789762065.181054:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762065.181059:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762065.181091:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512913792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762065.181118:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762065.181122:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762065.181124:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762065.181129:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2039 00000100:00080000:3.0:1789762065.181135:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762065.181140:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762065.181146:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512913792:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789762065.181150:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762065.181169:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512913920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762065.181179:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762065.181182:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762065.181190:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762065.181194:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2039 00000100:00080000:3.0:1789762065.181197:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762065.181202:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789762065.181206:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789762065.181210:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762065.181223:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512914048/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762065.181231:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762065.181234:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762065.181236:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762065.181239:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2039 00000100:00080000:3.0:1789762065.181242:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762065.181243:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762065/real 1789762065] req@ffff8dc0107ae680 x1876699512913792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762065.181247:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762065.181257:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762065.181263:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512913792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762065.181269:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512914176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762065.181281:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762065.181286:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762065.181290:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512913792:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762065.181312:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512913920:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762065.181327:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762065.181336:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762065/real 1789762065] req@ffff8dc0107af800 x1876699512913920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762065.181343:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512913920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762065.181350:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762065.181352:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762065.181354:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512913920:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762065.181357:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512914048:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762065.181368:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762065.181374:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762065/real 1789762065] req@ffff8dc0107ac380 x1876699512914048/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762065.181381:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512914048/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762065.181386:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762065.181388:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762065.181390:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512914048:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762065.181393:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512914176:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762065.181403:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762065.181408:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762065/real 1789762065] req@ffff8dc0107adf80 x1876699512914176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762065.181415:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512914176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762065.181420:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762065.181422:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762065.181424:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512914176:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762066.205093:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762067.229014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762068.253075:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762069.277010:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762070.301031:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762070.301039:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762070.301042:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789762070.301052:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762070.301052:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2044 00000100:00080000:3.0:1789762070.301064:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762070.301071:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762070.301103:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512914304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762070.301123:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762070.301126:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762070.301132:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762070.301136:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2044 00000100:00080000:3.0:1789762070.301139:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762070.301142:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762070.301150:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762070.301155:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512914304:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762070.301163:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512914432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762070.301171:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762070.301173:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762070.301175:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762070.301178:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2044 00000100:00080000:3.0:1789762070.301181:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762070.301183:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762070.301186:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762070.301196:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512914560/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762070.301203:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762070.301206:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762070.301207:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762070.301210:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2044 00000100:00080000:3.0:1789762070.301213:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762070.301216:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789762070.301221:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789762070.301221:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762070.301231:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512914688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762070.301259:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762070/real 1789762070] req@ffff8dc0107af480 x1876699512914304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762070.301273:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512914304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762070.301281:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762070.301284:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762070.301286:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512914304:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762070.301304:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512914432:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762070.301318:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762070.301325:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762070/real 1789762070] req@ffff8dc0107ad180 x1876699512914432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762070.301333:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512914432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762070.301339:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762070.301341:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762070.301343:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512914432:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762070.301347:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512914560:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762070.301358:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762070.301364:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762070/real 1789762070] req@ffff8dc0107aed80 x1876699512914560/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762070.301372:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512914560/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762070.301377:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762070.301379:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762070.301382:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512914560:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762070.301385:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512914688:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762070.301395:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762070.301400:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762070/real 1789762070] req@ffff8dc0107ad500 x1876699512914688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762070.301408:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512914688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762070.301413:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762070.301414:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762070.301416:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512914688:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762071.325095:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762072.349089:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762073.373021:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762074.397076:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762075.421036:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:1.0:1789762075.421038:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762075.421050:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762075.421057:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762075.421069:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2050 00000100:00080000:2.0:1789762075.421089:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762075.421098:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762075.421145:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512914816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762075.421176:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762075.421180:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762075.421182:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762075.421188:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2050 00000100:00080000:2.0:1789762075.421193:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762075.421198:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762075.421211:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762075.421217:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512914816:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762075.421232:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512914944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762075.421255:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762075.421260:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762075.421262:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762075.421267:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2050 00000100:00080000:2.0:1789762075.421272:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762075.421276:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762075.421281:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789762075.421294:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762075.421298:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512915072/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762075.421314:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762075.421318:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762075.421321:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762075.421326:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2050 00000100:00080000:2.0:1789762075.421329:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762075.421334:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762075.421339:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762075.421352:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512915200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762075.421357:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762075/real 1789762075] req@ffff8dc00368f480 x1876699512914816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762075.421414:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512914816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762075.421426:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762075.421431:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762075.421435:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512914816:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762075.421473:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512914944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762075.421495:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762075.421507:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762075/real 1789762075] req@ffff8dc00368fb80 x1876699512914944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762075.421521:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512914944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762075.421530:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762075.421533:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762075.421537:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512914944:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762075.421544:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512915072:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762075.421567:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762075.421580:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762075/real 1789762075] req@ffff8dc00368d880 x1876699512915072/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762075.421594:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512915072/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762075.421610:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762075.421614:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762075.421618:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512915072:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762075.421624:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512915200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762075.421647:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762075.421660:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762075/real 1789762075] req@ffff8dc00368f100 x1876699512915200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762075.421672:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512915200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762075.421681:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762075.421684:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762075.421688:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512915200:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762076.445033:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762077.469055:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762078.493061:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762079.517036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762080.541019:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762080.541033:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762080.541038:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762080.541052:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2055 00000100:00080000:2.0:1789762080.541065:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762080.541075:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789762080.541127:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762080.541134:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512915328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762080.541166:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762080.541170:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762080.541172:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762080.541176:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2055 00000100:00080000:2.0:1789762080.541179:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762080.541183:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762080.541194:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762080.541207:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512915456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762080.541217:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512915328:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762080.541224:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762080.541229:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762080.541230:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762080.541234:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2055 00000100:00080000:2.0:1789762080.541236:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762080.541238:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762080.541242:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762080.541252:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512915584/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762080.541260:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762080.541263:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762080.541265:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762080.541267:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2055 00000100:00080000:2.0:1789762080.541269:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762080.541272:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762080.541275:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762080.541292:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512915712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762080.541359:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512915456:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762080.541396:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512915584:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762080.541445:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512915712:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762081.565067:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762082.589061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762083.613087:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789762083.613225:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:2.0:1789762083.613271:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789762083.613284:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789762083.613323:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789762083.613333:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789762083.613343:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:0.0:1789762083.613365:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762080/real 1789762083] req@ffff8dc00368d880 x1876699512915456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762083.613408:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512915456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762083.613419:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762083.613424:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762083.613428:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512915456:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762083.613436:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762080/real 1789762083] req@ffff8dc00368fb80 x1876699512915584/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762083.613448:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512915584/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762083.613456:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762083.613459:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762083.613463:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512915584:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762083.613470:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762080/real 1789762083] req@ffff8dc00368f480 x1876699512915712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762083.613482:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512915712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762083.613490:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762083.613493:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762083.613497:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512915712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762083.613504:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762080/real 1789762083] req@ffff8dc00368f100 x1876699512915328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762083.613518:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512915328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762083.613525:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762083.613527:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762083.613530:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512915328:192.168.203.156@tcp:250:kworker.0 00010000:00010000:2.0:1789762084.637038:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762085.661070:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762085.661071:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762085.661081:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762085.661086:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762085.661096:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2060 00000100:00080000:2.0:1789762085.661107:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762085.661116:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762085.661154:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512915840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762085.661184:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762085.661187:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762085.661189:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762085.661193:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2060 00000100:00080000:2.0:1789762085.661196:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762085.661200:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762085.661210:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762085.661227:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512915968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762085.661245:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512915840:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762085.661246:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762085.661261:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762085.661264:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762085.661271:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2060 00000100:00080000:2.0:1789762085.661278:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762085.661283:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762085.661290:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762085.661307:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512916096/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762085.661318:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762085.661322:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762085.661324:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762085.661329:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2060 00000100:00080000:2.0:1789762085.661334:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762085.661339:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762085.661346:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762085.661360:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512916224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762085.661367:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762085.661445:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762085/real 1789762085] req@ffff8dc00368ce00 x1876699512915840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762085.661472:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512915840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762085.661489:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762085.661494:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762085.661500:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512915840:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762085.661536:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512915968:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762085.661566:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762085.661581:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762085/real 1789762085] req@ffff8dc00368ea00 x1876699512915968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762085.661594:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512915968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762085.661610:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762085.661614:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762085.661618:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512915968:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762085.661626:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512916096:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762085.661649:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762085.661662:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762085/real 1789762085] req@ffff8dc00368c000 x1876699512916096/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762085.661673:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512916096/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762085.661682:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762085.661686:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762085.661689:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512916096:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762085.661696:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512916224:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762085.661719:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762085.661732:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762085/real 1789762085] req@ffff8dc00368ca80 x1876699512916224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762085.661742:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512916224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762085.661750:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762085.661753:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762085.661757:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512916224:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762086.685051:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762087.709100:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762088.733013:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762089.757094:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762090.781005:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762090.781015:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762090.781020:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762090.781031:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2065 00000100:00080000:2.0:1789762090.781042:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762090.781057:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789762090.781064:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762090.781101:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512916352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762090.781123:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762090.781127:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762090.781128:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762090.781132:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2065 00000100:00080000:2.0:1789762090.781135:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762090.781138:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762090.781160:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762090.781169:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512916352:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762090.781177:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512916480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762090.781191:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762090.781195:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762090.781197:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762090.781202:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2065 00000100:00080000:2.0:1789762090.781206:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762090.781211:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762090.781217:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762090.781228:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512916608/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762090.781236:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762090.781239:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762090.781240:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762090.781243:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2065 00000800:00000100:0.0:1789762090.781248:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762090.781248:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762090.781255:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762090.781260:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762090.781269:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512916736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762090.781300:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762090/real 1789762090] req@ffff8dc00368df80 x1876699512916352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762090.781317:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512916352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762090.781326:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762090.781351:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762090.781358:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512916352:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762090.781396:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512916480:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762090.781415:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762090.781431:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762090/real 1789762090] req@ffff8dc00368ed80 x1876699512916480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762090.781442:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512916480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762090.781448:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762090.781450:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762090.781453:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512916480:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762090.781456:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512916608:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762090.781470:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762090.781476:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762090/real 1789762090] req@ffff8dc00368e300 x1876699512916608/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762090.781484:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512916608/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762090.781499:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762090.781501:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762090.781503:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512916608:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762090.781507:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512916736:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762090.781517:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762090.781523:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762090/real 1789762090] req@ffff8dc00368f800 x1876699512916736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762090.781531:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512916736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762090.781536:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762090.781538:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762090.781539:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512916736:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762091.805116:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762092.829061:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762093.853105:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762094.877068:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762095.901057:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762095.901068:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762095.901073:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762095.901086:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2070 00000100:00080000:2.0:1789762095.901103:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762095.901112:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762095.901176:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512916864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762095.901227:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762095.901232:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762095.901236:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762095.901245:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2070 00000100:00080000:2.0:1789762095.901252:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762095.901265:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789762095.901283:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762095.901283:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512916864:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789762095.901308:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762095.901332:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512916992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762095.901357:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762095.901361:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762095.901363:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762095.901370:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2070 00000100:00080000:2.0:1789762095.901383:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789762095.901389:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762095.901390:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762095.901399:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762095.901409:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512917120/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762095.901420:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762095.901424:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762095.901426:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762095.901431:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2070 00000100:00080000:2.0:1789762095.901436:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762095.901441:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762095.901448:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762095.901466:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512917248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762095.901610:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762095/real 1789762095] req@ffff8dc00368c700 x1876699512916864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762095.901632:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512916864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762095.901643:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762095.901647:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762095.901651:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512916864:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762095.901766:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512916992:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762095.901807:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762095.901825:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762095/real 1789762095] req@ffff8dc00368dc00 x1876699512916992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762095.901843:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512916992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762095.901864:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762095.901869:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762095.901874:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512916992:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762095.901883:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512917120:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789762095.901921:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762095.901963:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762095/real 1789762095] req@ffff8dc00368e680 x1876699512917120/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762095.901979:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512917120/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762095.902005:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762095.902012:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762095.902018:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512917120:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789762095.902027:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512917248:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762095.902049:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762095.902060:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762095/real 1789762095] req@ffff8dc00368d180 x1876699512917248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762095.902070:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512917248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762095.902076:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762095.902079:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762095.902082:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512917248:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762096.925071:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762097.949002:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762098.973034:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762099.997131:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762101.021075:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1789762101.021096:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762101.021102:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762101.021110:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762101.021124:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2075 00000100:00080000:3.0:1789762101.021138:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762101.021148:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762101.021199:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512917376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762101.021249:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762101.021294:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762101.021297:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762101.021303:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2075 00000100:00080000:3.0:1789762101.021308:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762101.021314:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762101.021325:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512917376:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789762101.021326:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762101.021350:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512917504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762101.021385:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762101.021390:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762101.021393:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762101.021399:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2075 00000100:00080000:3.0:1789762101.021406:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762101.021414:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789762101.021421:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789762101.021427:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762101.021450:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512917632/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762101.021468:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762101.021474:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762101.021477:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762101.021493:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2075 00000100:00080000:3.0:1789762101.021500:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762101.021504:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762101/real 1789762101] req@ffff8dc0107ac000 x1876699512917376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762101.021511:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762101.021527:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762101.021537:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512917376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762101.021552:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512917760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762101.021554:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762101.021569:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762101.021583:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512917376:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762101.021619:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512917504:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762101.021648:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762101.021661:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762101/real 1789762101] req@ffff8dc0107ac700 x1876699512917504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762101.021673:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512917504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762101.021682:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762101.021685:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762101.021689:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512917504:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762101.021695:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512917632:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762101.021711:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762101.021720:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762101/real 1789762101] req@ffff8dc0107ae300 x1876699512917632/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762101.021756:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512917632/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762101.021767:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762101.021772:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762101.021776:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512917632:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762101.021783:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512917760:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762101.021807:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762101.021817:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762101/real 1789762101] req@ffff8dc0107aca80 x1876699512917760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762101.021826:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512917760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762101.021842:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762101.021844:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762101.021847:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512917760:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762102.045033:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762103.069040:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762104.093011:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762105.117025:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762106.140965:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762106.140977:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762106.140981:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762106.140993:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2080 00000100:00080000:3.0:1789762106.141003:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762106.141011:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789762106.141029:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762106.141051:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699512917888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762106.141078:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762106.141082:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762106.141085:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762106.141090:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2080 00000100:00080000:3.0:1789762106.141094:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762106.141099:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762106.141106:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512917888:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789762106.141109:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762106.141126:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699512918016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762106.141138:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762106.141142:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762106.141144:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762106.141148:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2080 00000800:00000100:0.0:1789762106.141161:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762106.141162:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762106.141170:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762106.141176:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762106.141191:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512918144/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762106.141203:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762106.141205:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762106/real 1789762106] req@ffff8dc0107adc00 x1876699512917888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762106.141209:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762106.141226:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762106.141236:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2080 00000100:00080000:3.0:1789762106.141245:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762106.141252:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762106.141258:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762106.141259:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699512917888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762106.141274:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789762106.141276:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512918272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762106.141283:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762106.141292:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512917888:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762106.141318:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512918016:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762106.141337:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762106.141349:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762106/real 1789762106] req@ffff8dc0107afb80 x1876699512918016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762106.141362:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699512918016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762106.141371:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762106.141374:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762106.141378:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512918016:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762106.141384:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512918144:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762106.141404:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762106.141413:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762106/real 1789762106] req@ffff8dc0107aca80 x1876699512918144/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762106.141421:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512918144/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762106.141430:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762106.141432:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762106.141435:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512918144:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762106.141438:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512918272:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762106.141449:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762106.141455:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762106/real 1789762106] req@ffff8dc0107ae300 x1876699512918272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762106.141462:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512918272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762106.141467:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762106.141469:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762106.141471:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512918272:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762107.165025:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762108.189062:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762109.213232:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762110.237182:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762111.260995:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762111.261004:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762111.261008:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789762111.261020:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762111.261021:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2085 00000100:00080000:2.0:1789762111.261037:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762111.261046:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762111.261081:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512918400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762111.261108:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762111.261110:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762111.261112:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762111.261115:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2085 00000100:00080000:2.0:1789762111.261118:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762111.261121:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762111.261129:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762111.261139:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512918528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762111.261149:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762111.261154:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789762111.261155:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512918400:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762111.261157:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762111.261170:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2085 00000100:00080000:2.0:1789762111.261174:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762111.261192:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762111.261197:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762111.261214:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512918656/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762111.261223:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762111.261225:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762111.261227:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762111.261232:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2085 00000100:00080000:2.0:1789762111.261236:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762111.261239:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762111.261242:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762111.261253:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512918784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762111.261272:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762111.261332:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762111/real 1789762111] req@ffff8dc00368d180 x1876699512918400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762111.261351:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512918400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762111.261368:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762111.261371:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762111.261375:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512918400:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762111.261402:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512918528:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762111.261422:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762111.261432:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762111/real 1789762111] req@ffff8dc00368e680 x1876699512918528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762111.261442:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512918528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762111.261449:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762111.261452:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762111.261455:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512918528:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762111.261459:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512918656:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762111.261473:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762111.261480:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762111/real 1789762111] req@ffff8dc00368dc00 x1876699512918656/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762111.261489:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512918656/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762111.261495:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762111.261506:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762111.261509:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512918656:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762111.261515:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512918784:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762111.261529:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762111.261537:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762111/real 1789762111] req@ffff8dc00368c700 x1876699512918784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762111.261547:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512918784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762111.261553:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762111.261556:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762111.261558:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512918784:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762112.285123:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762113.309045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762114.333011:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762115.357115:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762116.381061:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762116.381069:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762116.381073:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1789762116.381078:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762116.381082:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2090 00000100:00080000:3.0:1789762116.381092:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762116.381098:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762116.381157:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512918912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762116.381188:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762116.381191:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762116.381193:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762116.381195:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2090 00000100:00080000:3.0:1789762116.381198:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762116.381201:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762116.381208:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762116.381218:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512919040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762116.381230:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762116.381234:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762116.381237:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762116.381238:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512918912:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762116.381243:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2090 00000100:00080000:3.0:1789762116.381247:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762116.381250:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762116.381253:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762116.381263:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512919168/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762116.381270:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762116.381273:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762116.381274:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762116.381277:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2090 00000100:00080000:3.0:1789762116.381280:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762116.381282:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762116.381285:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762116.381293:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512919296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762116.381300:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762116.381344:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762116/real 1789762116] req@ffff8dc0107ac700 x1876699512918912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762116.381360:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512918912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762116.381367:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762116.381370:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762116.381373:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512918912:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762116.381389:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512919040:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762116.381406:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762116.381414:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762116/real 1789762116] req@ffff8dc0107ac000 x1876699512919040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762116.381422:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512919040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762116.381427:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762116.381429:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762116.381431:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512919040:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762116.381435:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512919168:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762116.381445:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762116.381451:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762116/real 1789762116] req@ffff8dc0107ad500 x1876699512919168/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762116.381458:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512919168/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762116.381464:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762116.381466:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762116.381467:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512919168:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762116.381471:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512919296:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762116.381481:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762116.381487:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762116/real 1789762116] req@ffff8dc0107aed80 x1876699512919296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762116.381499:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512919296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762116.381504:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762116.381506:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762116.381508:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512919296:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762117.405051:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762118.429158:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762119.453068:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762120.477046:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762121.501043:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762121.501062:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762121.501066:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762121.501074:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2096 00000100:00080000:3.0:1789762121.501082:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762121.501087:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789762121.501110:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762121.501149:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512919424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762121.501169:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762121.501172:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762121.501174:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762121.501177:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2096 00000100:00080000:3.0:1789762121.501180:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762121.501184:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762121.501192:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762121.501202:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512919424:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762121.501209:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512919552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762121.501222:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762121.501226:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762121.501227:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762121.501232:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2096 00000100:00080000:3.0:1789762121.501246:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762121.501250:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762121.501256:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762121.501270:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512919680/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762121.501275:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762121.501282:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762121.501285:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762121.501287:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762121.501290:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2096 00000100:00080000:3.0:1789762121.501294:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762121.501297:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762121.501300:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762121.501308:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512919808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762121.501319:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762121/real 1789762121] req@ffff8dc0107ad180 x1876699512919424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762121.501335:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512919424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762121.501344:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762121.501347:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762121.501350:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512919424:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762121.501378:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512919552:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762121.501394:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762121.501403:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762121/real 1789762121] req@ffff8dc0107af480 x1876699512919552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762121.501411:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512919552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762121.501417:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762121.501420:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762121.501423:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512919552:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762121.501428:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512919680:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762121.501447:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762121.501459:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762121/real 1789762121] req@ffff8dc0107adf80 x1876699512919680/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762121.501472:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512919680/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762121.501482:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762121.501484:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762121.501486:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512919680:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762121.501491:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512919808:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762121.501506:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762121.501513:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762121/real 1789762121] req@ffff8dc0107ac380 x1876699512919808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762121.501521:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512919808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762121.501527:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762121.501529:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762121.501531:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512919808:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762122.525072:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762123.549023:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762124.573199:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762125.597018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762126.620983:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762126.620992:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762126.620995:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789762126.621000:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762126.621004:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2101 00000100:00080000:3.0:1789762126.621013:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762126.621019:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762126.621051:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512919936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762126.621087:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762126.621089:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762126.621091:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762126.621094:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2101 00000100:00080000:3.0:1789762126.621096:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762126.621098:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762126.621106:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762126.621115:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512920064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762126.621122:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762126.621124:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762126.621126:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762126.621128:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2101 00000100:00080000:3.0:1789762126.621130:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762126.621134:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762126.621137:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762126.621143:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512919936:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762126.621147:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512920192/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762126.621158:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762126.621160:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762126.621162:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762126.621164:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2101 00000100:00080000:3.0:1789762126.621167:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762126.621169:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762126.621171:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762126.621178:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512920320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762126.621213:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762126.621255:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762126/real 1789762126] req@ffff8dc0107af800 x1876699512919936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762126.621268:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512919936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762126.621277:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762126.621280:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762126.621284:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512919936:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762126.621291:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512920064:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762126.621305:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762126.621314:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762126/real 1789762126] req@ffff8dc0107ae680 x1876699512920064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762126.621322:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512920064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762126.621327:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762126.621329:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762126.621331:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512920064:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762126.621348:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512920192:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762126.621360:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762126.621366:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762126/real 1789762126] req@ffff8dc0107ace00 x1876699512920192/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762126.621374:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512920192/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762126.621379:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762126.621381:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762126.621382:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512920192:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762126.621386:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512920320:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762126.621396:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762126.621402:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762126/real 1789762126] req@ffff8dc0107af100 x1876699512920320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762126.621409:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512920320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762126.621415:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762126.621416:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762126.621418:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512920320:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762127.645163:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762128.669191:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762129.693016:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762130.717098:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762131.741004:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762131.741015:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762131.741020:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762131.741042:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2106 00000100:00080000:3.0:1789762131.741052:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762131.741061:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762131.741097:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512920448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1789762131.741098:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762131.741152:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762131.741157:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762131.741160:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762131.741167:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2106 00000100:00080000:3.0:1789762131.741172:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762131.741186:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512920448:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762131.741189:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762131.741206:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762131.741226:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512920576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762131.741257:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762131.741261:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762131.741263:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762131.741268:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2106 00000100:00080000:3.0:1789762131.741272:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762131.741279:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789762131.741282:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789762131.741285:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762131.741305:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512920704/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762131.741314:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762131.741317:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762131.741319:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762131.741329:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762131/real 1789762131] req@ffff8dc0107aea00 x1876699512920448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762131.741349:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512920448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762131.741360:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762131.741365:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762131.741369:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512920448:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762131.741379:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2106 00000100:00080000:3.0:1789762131.741384:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762131.741387:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762131.741391:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762131.741397:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512920576:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762131.741404:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512920832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762131.741434:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762131.741447:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762131/real 1789762131] req@ffff8dc0107ad880 x1876699512920576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762131.741473:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512920576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762131.741493:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762131.741496:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762131.741500:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512920576:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762131.741507:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512920704:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762131.741530:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762131.741543:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762131/real 1789762131] req@ffff8dc0107af100 x1876699512920704/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762131.741554:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512920704/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762131.741565:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762131.741568:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762131.741570:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512920704:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762131.741582:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512920832:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762131.741599:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762131.741627:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762131/real 1789762131] req@ffff8dc0107ace00 x1876699512920832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762131.741642:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512920832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762131.741653:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762131.741658:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762131.741662:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512920832:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762132.765034:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762133.789084:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762134.813076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762135.837009:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762136.861060:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762136.861072:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762136.861076:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762136.861089:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2111 00010000:00010000:1.0:1789762136.861101:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762136.861101:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762136.861110:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762136.861190:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512920960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762136.861227:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762136.861241:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762136.861248:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762136.861255:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2111 00000100:00080000:3.0:1789762136.861261:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762136.861267:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762136.861281:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762136.861292:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512920960:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762136.861320:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512921088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762136.861337:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762136.861341:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762136.861344:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762136.861349:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2111 00000100:00080000:3.0:1789762136.861354:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762136.861359:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762136.861367:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762136.861394:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512921216/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762136.861401:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762136.861404:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762136.861405:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762136.861408:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2111 00000100:00080000:3.0:1789762136.861418:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762136.861421:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762136.861424:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762136.861433:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512921344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762136.861434:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762136.861555:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762136/real 1789762136] req@ffff8dc0107ae680 x1876699512920960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762136.861579:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512920960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762136.861591:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762136.861597:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762136.861602:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512920960:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762136.861629:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512921088:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762136.861655:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762136.861669:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762136/real 1789762136] req@ffff8dc0107af800 x1876699512921088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762136.861705:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512921088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762136.861719:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762136.861724:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762136.861731:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512921088:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762136.861742:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512921216:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762136.861798:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762136.861810:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762136/real 1789762136] req@ffff8dc0107ac380 x1876699512921216/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762136.861822:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512921216/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762136.861831:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762136.861834:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762136.861838:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512921216:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762136.861844:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512921344:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762136.861864:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762136.861875:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762136/real 1789762136] req@ffff8dc0107adf80 x1876699512921344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762136.861888:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512921344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762136.861896:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762136.861900:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762136.861903:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512921344:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762137.885190:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762138.909107:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762139.933102:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762140.957099:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762141.981046:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762141.981057:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762141.981061:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762141.981074:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2116 00000100:00080000:1.0:1789762141.981085:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762141.981093:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789762141.981106:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789762141.981140:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512921472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762141.981164:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762141.981167:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762141.981168:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762141.981172:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2116 00000100:00080000:1.0:1789762141.981176:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762141.981179:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762141.981189:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762141.981197:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512921472:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762141.981201:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512921600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762141.981212:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762141.981216:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762141.981217:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762141.981220:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2116 00000100:00080000:1.0:1789762141.981222:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762141.981225:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762141.981228:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762141.981237:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512921728/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762141.981245:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762141.981248:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762141.981250:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762141.981254:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2116 00000100:00080000:1.0:1789762141.981267:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762141.981273:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762141.981279:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762141.981282:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762141.981299:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512921856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762141.981337:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762141/real 1789762141] req@ffff8dc005c94e00 x1876699512921472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762141.981355:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512921472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762141.981364:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762141.981367:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762141.981371:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512921472:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762141.981396:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512921600:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762141.981412:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762141.981421:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762141/real 1789762141] req@ffff8dc005c96300 x1876699512921600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762141.981435:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512921600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762141.981440:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762141.981443:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762141.981445:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512921600:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762141.981448:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512921728:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762141.981460:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762141.981466:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762141/real 1789762141] req@ffff8dc005c97b80 x1876699512921728/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762141.981474:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512921728/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762141.981479:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762141.981481:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762141.981483:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512921728:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762141.981486:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512921856:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762141.981497:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762141.981503:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762141/real 1789762141] req@ffff8dc005c94380 x1876699512921856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762141.981510:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512921856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762141.981520:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762141.981522:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762141.981524:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512921856:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762143.005068:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762144.029089:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762145.053048:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762146.077130:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762147.101015:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762147.101025:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762147.101031:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762147.101046:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2121 00000100:00080000:1.0:1789762147.101060:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762147.101069:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789762147.101079:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789762147.101123:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512921984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762147.101155:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762147.101160:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762147.101162:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762147.101169:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2121 00000100:00080000:1.0:1789762147.101173:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762147.101179:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762147.101188:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512921984:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1789762147.101190:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762147.101207:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512922112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762147.101220:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762147.101225:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762147.101227:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762147.101233:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2121 00000100:00080000:1.0:1789762147.101246:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762147.101251:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762147.101258:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762147.101272:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512922240/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762147.101285:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762147.101289:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762147.101292:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762147.101297:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2121 00000100:00100000:0.0:1789762147.101302:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512922112:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1789762147.101303:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762147.101312:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762147.101319:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762147.101333:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512922368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762147.101340:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512922240:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762147.101371:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512922368:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762148.125092:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762149.149051:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762150.173236:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762150.173271:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:1.0:1789762150.173314:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762150.173327:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762150.173359:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762150.173370:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762150.173378:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:0.0:1789762150.173433:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762147/real 1789762150] req@ffff8dc005c94700 x1876699512922368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762150.173475:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512922368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762150.173491:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762150.173498:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762150.173504:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512922368:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762150.173515:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762147/real 1789762150] req@ffff8dc005c95500 x1876699512922112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762150.173532:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512922112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762150.173542:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762150.173547:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762150.173551:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512922112:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762150.173566:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762147/real 1789762150] req@ffff8dc005c95f80 x1876699512922240/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762150.173581:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512922240/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762150.173590:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762150.173594:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762150.173597:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512922240:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762150.173606:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762147/real 1789762150] req@ffff8dc005c96a00 x1876699512921984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762150.173619:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512921984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762150.173627:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762150.173630:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762150.173634:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512921984:192.168.203.156@tcp:250:kworker.0 00010000:00010000:2.0:1789762151.197085:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762152.221043:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762152.221062:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762152.221070:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762152.221074:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762152.221084:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2126 00000100:00080000:2.0:1789762152.221093:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762152.221101:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762152.221134:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512922496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762152.221161:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762152.221165:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762152.221167:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762152.221171:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2126 00000100:00080000:2.0:1789762152.221175:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762152.221179:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762152.221188:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762152.221202:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512922624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762152.221208:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512922496:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762152.221227:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762152.221232:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762152.221235:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762152.221240:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2126 00000100:00080000:2.0:1789762152.221244:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762152.221248:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762152.221254:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762152.221266:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512922752/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762152.221277:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789762152.221278:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762152.221284:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762152.221287:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762152.221292:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2126 00000100:00080000:2.0:1789762152.221303:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762152.221307:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762152.221313:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762152.221321:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762152/real 1789762152] req@ffff8dc00368f800 x1876699512922496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762152.221326:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512922880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762152.221360:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512922496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762152.221374:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762152.221379:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762152.221383:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512922496:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762152.221408:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512922624:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762152.221450:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762152.221464:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762152/real 1789762152] req@ffff8dc00368e300 x1876699512922624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762152.221476:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512922624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762152.221485:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762152.221488:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762152.221492:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512922624:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762152.221498:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512922752:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762152.221519:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762152.221532:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762152/real 1789762152] req@ffff8dc00368ed80 x1876699512922752/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762152.221544:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512922752/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762152.221552:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762152.221556:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762152.221559:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512922752:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762152.221565:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512922880:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762152.221586:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762152.221609:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762152/real 1789762152] req@ffff8dc00368df80 x1876699512922880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762152.221621:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512922880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762152.221628:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762152.221631:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762152.221634:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512922880:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762153.245086:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762154.269083:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762155.293073:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762156.317012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762157.341012:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762157.341023:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762157.341027:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762157.341037:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2131 00000100:00080000:2.0:1789762157.341047:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762157.341054:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762157.341093:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512923008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1789762157.341112:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762157.341120:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762157.341134:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762157.341138:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762157.341144:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2131 00000100:00080000:2.0:1789762157.341148:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762157.341150:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512923008:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762157.341156:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762157.341167:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762157.341183:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512923136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762157.341196:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762157.341200:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762157.341202:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789762157.341209:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762157.341209:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2131 00000100:00080000:2.0:1789762157.341216:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762157.341222:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762157.341237:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762157.341246:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762157/real 1789762157] req@ffff8dc00368ca80 x1876699512923008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762157.341253:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512923264/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762157.341265:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512923008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762157.341269:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762157.341278:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762157.341281:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762157.341283:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762157.341288:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2131 00000100:00080000:0.0:1789762157.341289:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762157.341295:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512923008:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762157.341307:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762157.341315:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762157.341316:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512923136:192.168.203.156@tcp:38:kworker.0 00000008:00100000:2.0:1789762157.341323:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762157.341337:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762157.341343:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512923392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762157.341352:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762157/real 1789762157] req@ffff8dc00368c000 x1876699512923136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762157.341386:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512923136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762157.341396:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762157.341399:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762157.341401:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512923136:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762157.341407:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512923264:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762157.341432:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762157.341441:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762157/real 1789762157] req@ffff8dc00368ea00 x1876699512923264/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762157.341449:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512923264/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762157.341454:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762157.341456:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762157.341458:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512923264:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762157.341467:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512923392:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762157.341486:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762157.341493:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762157/real 1789762157] req@ffff8dc00368ce00 x1876699512923392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762157.341501:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512923392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762157.341510:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762157.341512:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762157.341514:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512923392:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762158.365051:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762159.389103:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762160.413108:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762161.437098:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762162.461011:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762162.461020:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1789762162.461020:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762162.461024:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762162.461033:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2137 00000100:00080000:0.0:1789762162.461041:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762162.461046:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762162.461076:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b800 x1876699512923520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762162.461099:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762162.461102:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762162.461104:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762162.461109:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2137 00000100:00080000:0.0:1789762162.461112:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762162.461116:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762162.461125:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762162.461142:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149f80 x1876699512923648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762162.461144:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512923520:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789762162.461159:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762162.461162:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762162.461163:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762162.461168:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2137 00000100:00080000:0.0:1789762162.461171:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762162.461176:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762162.461182:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762162.461196:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149180 x1876699512923776/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762162.461208:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762162.461211:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762162.461213:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762162.461217:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2137 00000100:00080000:0.0:1789762162.461221:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762162.461224:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762162.461228:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762162.461236:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149880 x1876699512923904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762162.461326:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762162.461405:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762162/real 1789762162] req@ffff8dc00514b800 x1876699512923520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762162.461424:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b800 x1876699512923520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762162.461447:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762162.461452:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762162.461455:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512923520:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762162.461483:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512923648:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762162.461519:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762162.461529:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762162/real 1789762162] req@ffff8dc005149f80 x1876699512923648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762162.461537:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149f80 x1876699512923648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762162.461543:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762162.461545:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762162.461547:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512923648:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762162.461558:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512923776:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789762162.461571:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762162.461578:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762162/real 1789762162] req@ffff8dc005149180 x1876699512923776/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762162.461586:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149180 x1876699512923776/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762162.461591:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762162.461594:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762162.461596:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512923776:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789762162.461600:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512923904:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762162.461614:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762162.461643:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762162/real 1789762162] req@ffff8dc005149880 x1876699512923904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762162.461651:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149880 x1876699512923904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762162.461657:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762162.461659:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762162.461661:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512923904:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762163.485106:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762164.509188:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762165.533150:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762166.557047:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762167.581007:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762167.581031:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762167.581036:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762167.581048:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2142 00000100:00080000:0.0:1789762167.581059:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762167.581067:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762167.581113:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149c00 x1876699512924032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762167.581149:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762167.581153:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762167.581156:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762167.581160:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2142 00000100:00080000:0.0:1789762167.581162:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762167.581165:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762167.581176:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762167.581186:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b480 x1876699512924160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762167.581195:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762167.581198:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762167.581199:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762167.581202:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2142 00000100:00080000:0.0:1789762167.581205:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762167.581207:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762167.581211:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762167.581221:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512924032:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762167.581222:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514bb80 x1876699512924288/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762167.581237:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762167.581240:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762167.581242:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762167.581250:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2142 00000100:00080000:0.0:1789762167.581254:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762167.581258:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762167.581263:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762167.581276:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148380 x1876699512924416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762167.581311:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00010000:00010000:2.0:1789762167.581363:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762167.581485:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762167/real 1789762167] req@ffff8dc005149c00 x1876699512924032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762167.581513:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149c00 x1876699512924032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762167.581526:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762167.581530:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762167.581534:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512924032:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762167.581542:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512924160:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762167.581789:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762167.583185:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762167/real 1789762167] req@ffff8dc00514b480 x1876699512924160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762167.583228:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b480 x1876699512924160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762167.583238:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762167.583241:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762167.583245:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512924160:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762167.583286:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512924288:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789762167.583312:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762167.583326:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762167/real 1789762167] req@ffff8dc00514bb80 x1876699512924288/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762167.583337:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514bb80 x1876699512924288/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762167.583345:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762167.583348:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762167.583351:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512924288:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789762167.583356:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512924416:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762167.583375:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762167.583387:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762167/real 1789762167] req@ffff8dc005148380 x1876699512924416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762167.583405:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148380 x1876699512924416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762167.583413:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762167.583417:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762167.583420:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512924416:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762168.605027:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762169.629069:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762170.653022:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762171.677093:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762172.700999:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762172.701008:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762172.701012:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762172.701020:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2147 00000100:00080000:2.0:1789762172.701029:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762172.701034:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789762172.701054:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762172.701102:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512924544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762172.701133:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762172.701137:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762172.701138:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762172.701142:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2147 00000100:00080000:2.0:1789762172.701145:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762172.701149:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762172.701159:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762172.701170:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512924672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762172.701180:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512924544:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762172.701186:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762172.701191:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762172.701193:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762172.701198:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2147 00000100:00080000:2.0:1789762172.701201:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762172.701204:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762172.701209:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762172.701218:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512924800/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762172.701227:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762172.701229:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762172.701231:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762172.701234:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2147 00000100:00080000:2.0:1789762172.701263:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762172.701266:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762172.701269:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762172.701276:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512924928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762172.701308:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762172.701362:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762172/real 1789762172] req@ffff8dc00368f100 x1876699512924544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762172.701386:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512924544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762172.701396:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762172.701399:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762172.701402:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512924544:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762172.701428:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512924672:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762172.701447:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762172.701460:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762172/real 1789762172] req@ffff8dc00368f480 x1876699512924672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762172.701482:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512924672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762172.701492:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762172.701496:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762172.701500:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512924672:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762172.701521:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512924800:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789762172.701541:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762172.701551:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762172/real 1789762172] req@ffff8dc00368fb80 x1876699512924800/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762172.701560:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512924800/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762172.701582:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762172.701585:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762172.701587:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512924800:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789762172.701591:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512924928:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762172.701605:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762172.701612:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762172/real 1789762172] req@ffff8dc00368d880 x1876699512924928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762172.701620:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512924928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762172.701630:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762172.701633:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762172.701635:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512924928:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762173.725136:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762174.749162:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762175.773068:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762176.797099:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762177.821016:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1789762177.821017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762177.821029:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762177.821035:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762177.821048:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2152 00000100:00080000:3.0:1789762177.821059:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762177.821065:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762177.821113:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512925056/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762177.821146:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762177.821151:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762177.821153:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762177.821159:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2152 00000100:00080000:3.0:1789762177.821164:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762177.821168:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762177.821178:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762177.821180:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512925056:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762177.821194:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512925184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762177.821208:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762177.821213:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762177.821216:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762177.821220:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2152 00000100:00080000:3.0:1789762177.821225:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762177.821230:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762177.821236:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762177.821251:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512925312/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762177.821260:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762177.821272:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762177.821276:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762177.821279:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762177.821286:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2152 00000100:00080000:3.0:1789762177.821290:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762177.821296:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762177.821301:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762177.821314:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762177/real 1789762177] req@ffff8dc0107af480 x1876699512925056/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762177.821319:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512925440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762177.821342:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512925056/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762177.821352:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762177.821356:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762177.821360:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512925056:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762177.821382:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512925184:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762177.821402:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762177.821417:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762177/real 1789762177] req@ffff8dc0107ad180 x1876699512925184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762177.821431:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512925184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762177.821447:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762177.821450:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762177.821452:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512925184:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762177.821457:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512925312:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762177.821476:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762177.821485:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762177/real 1789762177] req@ffff8dc0107aed80 x1876699512925312/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762177.821493:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512925312/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762177.821499:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762177.821501:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762177.821503:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512925312:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762177.821507:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512925440:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762177.821518:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762177.821524:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762177/real 1789762177] req@ffff8dc0107ad500 x1876699512925440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762177.821531:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512925440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762177.821536:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762177.821538:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762177.821540:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512925440:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762178.845023:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762179.869097:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762180.893057:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762181.917218:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762182.941136:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762182.941151:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762182.941158:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762182.941177:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2157 00000100:00080000:1.0:1789762182.941195:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1789762182.941199:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762182.941209:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762182.941284:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512925568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762182.941333:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762182.941341:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762182.941345:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762182.941353:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2157 00000100:00080000:1.0:1789762182.941363:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762182.941375:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762182.941394:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762182.941399:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512925568:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762182.941429:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512925696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762182.941458:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762182.941467:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762182.941472:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762182.941480:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2157 00000100:00080000:1.0:1789762182.941490:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762182.941500:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789762182.941509:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1789762182.941519:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762182.941552:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512925824/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762182.941574:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762182.941582:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762182.941586:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762182.941598:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762182/real 1789762182] req@ffff8dc005c97800 x1876699512925568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762182.941600:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2157 00000100:00080000:1.0:1789762182.941617:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762182.941632:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762182.941664:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512925568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789762182.941673:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789762182.941687:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762182.941696:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762182.941703:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512925568:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762182.941712:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512925952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762182.941749:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512925696:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762182.941789:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762182.941807:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762182/real 1789762182] req@ffff8dc005c95c00 x1876699512925696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762182.941824:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512925696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762182.941835:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762182.941840:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762182.941844:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512925696:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762182.941852:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512925824:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762182.941878:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762182.941890:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762182/real 1789762182] req@ffff8dc005c94a80 x1876699512925824/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762182.941946:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512925824/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762182.941963:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762182.941970:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762182.941977:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512925824:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762182.941989:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512925952:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762182.942032:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762182.942055:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762182/real 1789762182] req@ffff8dc005c97100 x1876699512925952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762182.942072:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512925952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762182.942083:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762182.942087:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762182.942091:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512925952:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762183.965019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762184.989066:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762186.013056:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762187.037066:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762188.060993:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762188.061003:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762188.061009:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762188.061021:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2162 00000100:00080000:2.0:1789762188.061032:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762188.061040:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762188.061082:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512926080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1789762188.061089:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762188.061110:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762188.061115:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762188.061117:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762188.061123:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2162 00000100:00080000:2.0:1789762188.061128:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762188.061133:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762188.061141:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762188.061144:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512926080:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762188.061155:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512926208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762188.061164:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762188.061166:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762188.061168:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762188.061170:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2162 00000100:00080000:2.0:1789762188.061173:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762188.061175:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762188.061179:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762188.061187:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512926336/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762188.061199:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762188.061202:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762188.061203:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762188.061205:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2162 00000100:00080000:2.0:1789762188.061208:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762188.061212:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762188.061216:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762188.061221:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762188.061229:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512926464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762188.061252:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762188/real 1789762188] req@ffff8dc00368d880 x1876699512926080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762188.061267:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512926080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762188.061275:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762188.061278:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762188.061281:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512926080:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762188.061299:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512926208:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762188.061313:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762188.061320:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762188/real 1789762188] req@ffff8dc00368fb80 x1876699512926208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762188.061328:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512926208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762188.061333:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762188.061335:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762188.061338:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512926208:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762188.061341:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512926336:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762188.061351:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762188.061357:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762188/real 1789762188] req@ffff8dc00368f480 x1876699512926336/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762188.061364:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512926336/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762188.061373:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762188.061375:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762188.061377:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512926336:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762188.061381:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512926464:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762188.061390:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762188.061396:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762188/real 1789762188] req@ffff8dc00368f100 x1876699512926464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762188.061403:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512926464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762188.061408:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762188.061410:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762188.061412:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512926464:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762189.085116:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762190.109020:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762191.133098:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762192.156974:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762193.181002:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762193.181010:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:0.0:1789762193.181013:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762193.181014:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762193.181024:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2167 00000100:00080000:3.0:1789762193.181033:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762193.181039:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762193.181071:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512926592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762193.181094:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762193.181098:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762193.181100:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762193.181106:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2167 00000100:00080000:3.0:1789762193.181111:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762193.181115:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762193.181121:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512926592:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789762193.181126:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762193.181152:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512926720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762193.181161:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762193.181164:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762193.181166:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762193.181169:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2167 00000100:00080000:3.0:1789762193.181173:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762193.181177:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762193.181178:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762193.181185:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762193.181195:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512926848/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762193.181203:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762193.181206:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762193.181208:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762193.181211:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2167 00000100:00100000:0.0:1789762193.181215:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762193/real 1789762193] req@ffff8dc0107ac000 x1876699512926592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762193.181216:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762193.181224:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762193.181232:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762193.181239:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512926592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762193.181242:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512926976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762193.181276:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762193.181280:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762193.181283:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512926592:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762193.181300:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512926720:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762193.181315:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762193.181323:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762193/real 1789762193] req@ffff8dc0107ac700 x1876699512926720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762193.181331:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512926720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762193.181337:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762193.181339:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762193.181341:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512926720:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762193.181345:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512926848:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762193.181357:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762193.181363:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762193/real 1789762193] req@ffff8dc0107ae300 x1876699512926848/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762193.181370:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512926848/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762193.181375:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762193.181377:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762193.181379:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512926848:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762193.181383:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512926976:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762193.181393:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762193.181399:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762193/real 1789762193] req@ffff8dc0107aca80 x1876699512926976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762193.181406:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512926976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762193.181411:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762193.181413:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762193.181415:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512926976:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762194.205032:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762195.229013:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762196.253014:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762197.277219:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762198.301023:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762198.301032:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762198.301036:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762198.301045:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2172 00000100:00080000:2.0:1789762198.301063:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762198.301069:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789762198.301080:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762198.301105:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512927104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762198.301128:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762198.301131:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762198.301132:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762198.301135:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2172 00000100:00080000:2.0:1789762198.301138:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762198.301140:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762198.301148:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762198.301157:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512927232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762198.301165:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762198.301168:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762198.301171:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762198.301173:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512927104:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762198.301175:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2172 00000100:00080000:2.0:1789762198.301180:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762198.301185:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762198.301189:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762198.301196:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512927360/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762198.301203:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762198.301205:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762198.301206:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762198.301209:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2172 00000100:00080000:2.0:1789762198.301211:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762198.301213:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762198.301216:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762198.301222:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512927488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762198.301247:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762198.301285:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762198/real 1789762198] req@ffff8dc00368ce00 x1876699512927104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762198.301299:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512927104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762198.301307:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762198.301310:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762198.301313:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512927104:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762198.301331:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512927232:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762198.301346:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762198.301355:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762198/real 1789762198] req@ffff8dc00368ea00 x1876699512927232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762198.301363:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512927232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762198.301369:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762198.301371:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762198.301373:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512927232:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762198.301377:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512927360:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762198.301387:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762198.301393:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762198/real 1789762198] req@ffff8dc00368c000 x1876699512927360/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762198.301400:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512927360/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762198.301405:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762198.301407:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762198.301409:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512927360:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762198.301412:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512927488:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762198.301422:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762198.301428:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762198/real 1789762198] req@ffff8dc00368ca80 x1876699512927488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762198.301435:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512927488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762198.301440:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762198.301442:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762198.301444:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512927488:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762199.325009:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762200.349052:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762201.373044:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762202.397060:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762203.421037:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762203.421045:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:1.0:1789762203.421047:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762203.421049:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762203.421058:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2178 00000100:00080000:0.0:1789762203.421068:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762203.421075:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762203.421110:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148000 x1876699512927616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762203.421133:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762203.421136:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762203.421137:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762203.421140:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2178 00000100:00080000:0.0:1789762203.421142:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762203.421145:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762203.421152:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762203.421165:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514ad80 x1876699512927744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762203.421175:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762203.421181:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762203.421183:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789762203.421183:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512927616:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789762203.421199:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2178 00000100:00080000:0.0:1789762203.421205:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762203.421209:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762203.421216:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762203.421233:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148700 x1876699512927872/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762203.421246:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762203.421250:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762203.421253:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762203.421257:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2178 00000100:00080000:0.0:1789762203.421262:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762203.421268:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1789762203.421276:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:0.0:1789762203.421281:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762203.421300:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514aa00 x1876699512928000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762203.421336:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762203/real 1789762203] req@ffff8dc005148000 x1876699512927616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762203.421353:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148000 x1876699512927616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762203.421363:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762203.421366:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762203.421369:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512927616:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762203.421392:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512927744:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762203.421409:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762203.421419:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762203/real 1789762203] req@ffff8dc00514ad80 x1876699512927744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762203.421427:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514ad80 x1876699512927744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762203.421433:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762203.421436:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762203.421438:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512927744:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762203.421442:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512927872:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762203.421456:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762203.421463:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762203/real 1789762203] req@ffff8dc005148700 x1876699512927872/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762203.421479:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148700 x1876699512927872/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762203.421486:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762203.421504:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762203.421507:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512927872:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762203.421514:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512928000:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762203.421535:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762203.421551:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762203/real 1789762203] req@ffff8dc00514aa00 x1876699512928000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762203.421572:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514aa00 x1876699512928000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762203.421588:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762203.421594:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762203.421600:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512928000:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762204.445105:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762205.469032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762206.493087:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762207.517061:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762208.541046:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762208.541062:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762208.541066:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762208.541089:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2183 00000100:00080000:0.0:1789762208.541100:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762208.541108:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762208.541163:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148a80 x1876699512928128/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762208.541187:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762208.541190:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762208.541191:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762208.541194:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2183 00000100:00080000:0.0:1789762208.541197:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762208.541199:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762208.541212:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762208.541239:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514aa00 x1876699512928256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762208.541249:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762208.541253:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762208.541256:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762208.541261:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2183 00000100:00080000:0.0:1789762208.541266:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762208.541270:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762208.541277:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762208.541292:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148700 x1876699512928384/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762208.541295:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512928128:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789762208.541319:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762208.541322:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762208.541324:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762208.541329:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2183 00000100:00080000:0.0:1789762208.541333:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762208.541337:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762208.541341:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762208.541351:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514ad80 x1876699512928512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762208.541371:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00010000:00010000:0.0:1789762208.541410:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762208.541435:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762208/real 1789762208] req@ffff8dc005148a80 x1876699512928128/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762208.541466:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148a80 x1876699512928128/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762208.541479:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762208.541483:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762208.541486:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512928128:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762208.541491:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512928256:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762208.541509:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762208.541518:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762208/real 1789762208] req@ffff8dc00514aa00 x1876699512928256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762208.541526:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514aa00 x1876699512928256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762208.541531:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762208.541534:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762208.541536:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512928256:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762208.541556:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512928384:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762208.541568:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762208.541574:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762208/real 1789762208] req@ffff8dc005148700 x1876699512928384/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762208.541581:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148700 x1876699512928384/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762208.541587:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762208.541589:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762208.541591:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512928384:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762208.541595:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512928512:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762208.541605:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762208.541611:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762208/real 1789762208] req@ffff8dc00514ad80 x1876699512928512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762208.541619:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514ad80 x1876699512928512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762208.541623:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762208.541625:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762208.541627:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512928512:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762209.565039:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762210.589163:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762211.613065:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762212.637109:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762213.661029:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762213.661048:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762213.661054:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762213.661066:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2188 00000100:00080000:0.0:1789762213.661079:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762213.661089:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762213.661141:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148000 x1876699512928640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762213.661165:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762213.661167:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762213.661168:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762213.661171:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2188 00000100:00080000:0.0:1789762213.661174:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762213.661177:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762213.661187:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762213.661197:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148380 x1876699512928768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762213.661206:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762213.661208:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762213.661210:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762213.661212:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2188 00000100:00080000:0.0:1789762213.661214:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762213.661217:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762213.661221:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762213.661225:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512928640:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762213.661231:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514bb80 x1876699512928896/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762213.661239:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762213.661242:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762213.661244:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762213.661246:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2188 00000100:00080000:0.0:1789762213.661248:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762213.661251:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762213.661254:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762213.661262:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b480 x1876699512929024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762213.661326:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512928768:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762213.661354:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512928896:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762213.661370:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512929024:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762213.661436:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762214.685056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762215.709097:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762216.733065:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762216.733161:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:1.0:1789762216.733198:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762216.733212:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762216.733245:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762216.733253:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762216.733263:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1789762216.733290:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762213/real 1789762216] req@ffff8dc00514bb80 x1876699512928896/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762216.733317:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514bb80 x1876699512928896/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762216.733330:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762216.733335:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762216.733340:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512928896:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762216.733349:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762213/real 1789762216] req@ffff8dc00514b480 x1876699512929024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762216.733363:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b480 x1876699512929024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762216.733372:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762216.733376:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762216.733380:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512929024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762216.733399:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762213/real 1789762216] req@ffff8dc005148000 x1876699512928640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762216.733413:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148000 x1876699512928640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762216.733420:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762216.733424:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762216.733427:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512928640:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762216.733433:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762213/real 1789762216] req@ffff8dc005148380 x1876699512928768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762216.733448:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148380 x1876699512928768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762216.733455:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762216.733459:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762216.733462:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512928768:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762217.757030:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762218.781089:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762218.781100:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762218.781111:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762218.781117:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762218.781129:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2193 00000100:00080000:2.0:1789762218.781142:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762218.781151:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762218.781190:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512929152/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762218.781233:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762218.781237:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762218.781239:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762218.781244:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2193 00000100:00080000:2.0:1789762218.781248:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762218.781253:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762218.781265:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762218.781287:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512929280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762218.781299:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762218.781303:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762218.781305:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762218.781309:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2193 00000100:00080000:2.0:1789762218.781314:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762218.781319:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762218.781324:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762218.781336:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512929408/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762218.781347:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762218.781352:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762218.781354:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762218.781359:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2193 00000100:00080000:2.0:1789762218.781364:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762218.781369:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762218.781386:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762218.781403:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512929152:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762218.781403:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512929536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762218.781509:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762218.781575:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762218/real 1789762218] req@ffff8dc00368df80 x1876699512929152/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762218.781597:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512929152/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762218.781609:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762218.781615:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762218.781621:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512929152:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762218.781631:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512929280:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762218.781659:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762218.781674:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762218/real 1789762218] req@ffff8dc00368ed80 x1876699512929280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762218.781687:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512929280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762218.781694:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762218.781696:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762218.781699:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512929280:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762218.781704:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512929408:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762218.781733:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762218.781746:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762218/real 1789762218] req@ffff8dc00368e300 x1876699512929408/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762218.781759:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512929408/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762218.781775:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762218.781780:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762218.781783:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512929408:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762218.782152:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512929536:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762218.782183:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762218.782199:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762218/real 1789762218] req@ffff8dc00368f800 x1876699512929536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762218.782219:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512929536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762218.782229:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762218.782233:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762218.782236:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512929536:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762219.805122:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762220.829029:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762221.853010:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762222.877041:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762223.901006:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762223.901017:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762223.901021:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762223.901035:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2198 00010000:00010000:0.0:1789762223.901037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762223.901051:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762223.901068:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762223.901118:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512929664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762223.901150:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762223.901153:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762223.901154:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762223.901157:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2198 00000100:00080000:2.0:1789762223.901161:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762223.901164:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762223.901174:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762223.901185:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512929792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762223.901195:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762223.901201:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789762223.901203:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512929664:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762223.901203:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762223.901209:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2198 00000100:00080000:2.0:1789762223.901216:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762223.901223:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762223.901227:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762223.901236:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512929920/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762223.901243:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762223.901246:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762223.901249:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762223.901253:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2198 00000100:00080000:2.0:1789762223.901258:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762223.901261:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762223.901266:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762223.901273:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512930048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762223.901291:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762223.901359:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762223/real 1789762223] req@ffff8dc00368c700 x1876699512929664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762223.901381:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512929664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762223.901393:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762223.901398:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762223.901403:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512929664:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762223.901428:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512929792:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762223.901452:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762223.901465:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762223/real 1789762223] req@ffff8dc00368dc00 x1876699512929792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762223.901478:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512929792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762223.901487:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762223.901490:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762223.901494:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512929792:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762223.901501:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512929920:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762223.901520:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762223.901530:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762223/real 1789762223] req@ffff8dc00368e680 x1876699512929920/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762223.901543:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512929920/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762223.901552:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762223.901555:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762223.901559:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512929920:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762223.901566:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512930048:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762223.901584:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762223.901595:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762223/real 1789762223] req@ffff8dc00368d180 x1876699512930048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762223.901608:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512930048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762223.901617:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762223.901620:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762223.901623:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512930048:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762224.925061:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762225.949097:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762226.973005:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762227.997080:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762229.020972:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762229.020980:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762229.020984:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762229.020992:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2203 00000100:00080000:2.0:1789762229.021007:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762229.021012:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789762229.021046:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762229.021051:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512930176/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762229.021078:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762229.021082:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762229.021084:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762229.021089:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2203 00000100:00080000:2.0:1789762229.021094:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762229.021099:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762229.021109:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762229.021113:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512930176:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762229.021125:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512930304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762229.021134:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762229.021137:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762229.021138:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762229.021141:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2203 00000100:00080000:2.0:1789762229.021145:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762229.021147:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762229.021151:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762229.021159:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512930432/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762229.021166:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762229.021168:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762229.021170:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762229.021173:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2203 00000800:00000100:0.0:1789762229.021178:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762229.021178:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762229.021182:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762229.021187:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762229.021196:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512930560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762229.021225:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762229/real 1789762229] req@ffff8dc00368d180 x1876699512930176/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762229.021254:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512930176/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762229.021265:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762229.021270:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762229.021274:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512930176:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762229.021306:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512930304:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762229.021329:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762229.021341:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762229/real 1789762229] req@ffff8dc00368e680 x1876699512930304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762229.021355:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512930304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762229.021364:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762229.021367:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762229.021371:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512930304:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762229.021377:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512930432:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762229.021394:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762229.021404:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762229/real 1789762229] req@ffff8dc00368dc00 x1876699512930432/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762229.021416:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512930432/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762229.021425:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762229.021427:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762229.021431:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512930432:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762229.021437:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512930560:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762229.021453:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762229.021462:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762229/real 1789762229] req@ffff8dc00368c700 x1876699512930560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762229.021474:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512930560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762229.021490:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762229.021493:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762229.021497:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512930560:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762230.045058:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762231.069045:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762232.093073:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762233.117098:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762234.141017:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762234.141028:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762234.141033:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762234.141045:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2208 00000100:00080000:2.0:1789762234.141055:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762234.141063:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789762234.141070:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762234.141126:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512930688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762234.141162:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762234.141168:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762234.141171:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762234.141179:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2208 00000100:00080000:2.0:1789762234.141187:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762234.141195:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762234.141210:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762234.141212:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512930688:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762234.141240:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512930816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762234.141278:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762234.141284:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762234.141304:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762234.141310:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2208 00000100:00080000:2.0:1789762234.141316:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762234.141322:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762234.141330:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789762234.141342:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762234.141350:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512930944/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762234.141368:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762234.141374:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762234.141376:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762234.141382:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2208 00000100:00080000:2.0:1789762234.141388:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762234.141394:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762234.141402:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762234.141404:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762234/real 1789762234] req@ffff8dc00368f800 x1876699512930688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762234.141422:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512931072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762234.141434:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512930688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762234.141454:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762234.141459:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762234.141464:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512930688:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762234.141504:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512930816:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762234.141533:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762234.141546:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762234/real 1789762234] req@ffff8dc00368e300 x1876699512930816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762234.141559:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512930816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762234.141573:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762234.141577:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762234.141580:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512930816:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762234.141586:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512930944:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762234.141603:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762234.141613:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762234/real 1789762234] req@ffff8dc00368ed80 x1876699512930944/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762234.141624:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512930944/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762234.141632:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762234.141635:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762234.141638:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512930944:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762234.141662:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512931072:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762234.141682:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762234.141698:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762234/real 1789762234] req@ffff8dc00368df80 x1876699512931072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762234.141717:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512931072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762234.141729:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762234.141735:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762234.141740:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512931072:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762235.165114:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762236.189036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762237.213207:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762238.237015:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762239.261026:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762239.261070:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762239.261077:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762239.261081:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762239.261094:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2213 00000100:00080000:0.0:1789762239.261106:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762239.261113:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762239.261157:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149c00 x1876699512931200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762239.261182:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762239.261185:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762239.261186:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762239.261189:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2213 00000100:00080000:0.0:1789762239.261191:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762239.261194:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762239.261205:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762239.261217:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149880 x1876699512931328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762239.261242:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762239.261244:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512931200:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789762239.261246:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762239.261251:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762239.261266:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2213 00000100:00080000:0.0:1789762239.261270:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762239.261274:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762239.261282:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762239.261295:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149180 x1876699512931456/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762239.261303:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762239.261307:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762239.261309:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762239.261314:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2213 00000100:00080000:0.0:1789762239.261320:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762239.261326:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1789762239.261334:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:0.0:1789762239.261335:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762239.261359:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149f80 x1876699512931584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762239.261382:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762239/real 1789762239] req@ffff8dc005149c00 x1876699512931200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762239.261401:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149c00 x1876699512931200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762239.261410:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762239.261414:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762239.261416:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512931200:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762239.261437:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512931328:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762239.261455:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762239.261467:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762239/real 1789762239] req@ffff8dc005149880 x1876699512931328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762239.261475:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149880 x1876699512931328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762239.261481:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762239.261483:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762239.261486:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512931328:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762239.261491:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512931456:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762239.261503:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762239.261509:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762239/real 1789762239] req@ffff8dc005149180 x1876699512931456/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762239.261517:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149180 x1876699512931456/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762239.261523:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762239.261524:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762239.261526:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512931456:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762239.261531:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512931584:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762239.261546:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762239.261552:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762239/real 1789762239] req@ffff8dc005149f80 x1876699512931584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762239.261562:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149f80 x1876699512931584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762239.261568:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762239.261571:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762239.261573:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512931584:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762240.285120:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762241.310307:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762242.333026:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762243.357124:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762244.381045:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762244.381064:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762244.381071:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1789762244.381108:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762244.381110:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2218 00000100:00080000:2.0:1789762244.381126:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762244.381137:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762244.381204:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512931712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762244.381246:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762244.381250:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762244.381252:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762244.381256:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2218 00000100:00080000:2.0:1789762244.381261:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762244.381265:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762244.381277:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762244.381291:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512931840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762244.381302:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762244.381306:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512931712:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762244.381307:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762244.381312:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762244.381323:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2218 00000100:00080000:2.0:1789762244.381327:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762244.381330:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762244.381335:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762244.381348:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512931968/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762244.381361:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762244.381365:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762244.381367:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762244.381370:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2218 00000100:00080000:2.0:1789762244.381374:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762244.381377:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762244.381381:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762244.381389:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512932096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762244.381418:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762244.381477:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762244/real 1789762244] req@ffff8dc00368ca80 x1876699512931712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762244.381498:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512931712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762244.381525:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762244.381529:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762244.381533:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512931712:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762244.381559:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512931840:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762244.381580:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762244.381592:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762244/real 1789762244] req@ffff8dc00368c000 x1876699512931840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762244.381603:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512931840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762244.381611:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762244.381614:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762244.381617:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512931840:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762244.381622:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512931968:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762244.381645:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762244.381660:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762244/real 1789762244] req@ffff8dc00368ea00 x1876699512931968/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762244.381678:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512931968/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762244.381691:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762244.381696:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762244.381700:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512931968:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762244.381711:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512932096:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762244.381733:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762244.381744:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762244/real 1789762244] req@ffff8dc00368ce00 x1876699512932096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762244.381755:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512932096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762244.381762:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762244.381765:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762244.381768:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512932096:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762245.405049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762246.429096:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762247.453090:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762248.477031:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762249.501011:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762249.501019:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762249.501023:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762249.501033:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2224 00000100:00080000:2.0:1789762249.501043:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762249.501049:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789762249.501057:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762249.501084:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512932224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762249.501107:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762249.501109:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762249.501110:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762249.501114:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2224 00000100:00080000:2.0:1789762249.501117:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762249.501121:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762249.501130:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762249.501141:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512932352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762249.501161:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762249.501166:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789762249.501168:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512932224:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762249.501168:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762249.501176:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2224 00000100:00080000:2.0:1789762249.501187:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762249.501191:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762249.501194:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762249.501205:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512932480/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762249.501213:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762249.501215:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762249.501216:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762249.501220:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2224 00000100:00080000:2.0:1789762249.501222:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762249.501226:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762249.501228:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762249.501235:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512932608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762249.501269:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762249.501313:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762249/real 1789762249] req@ffff8dc00368f100 x1876699512932224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762249.501328:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512932224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762249.501338:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762249.501341:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762249.501344:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512932224:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762249.501367:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512932352:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762249.501385:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762249.501394:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762249/real 1789762249] req@ffff8dc00368f480 x1876699512932352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762249.501402:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512932352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762249.501407:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762249.501410:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762249.501412:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512932352:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762249.501416:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512932480:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762249.501429:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762249.501435:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762249/real 1789762249] req@ffff8dc00368fb80 x1876699512932480/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762249.501443:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512932480/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762249.501448:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762249.501450:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762249.501452:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512932480:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762249.501456:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512932608:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762249.501467:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762249.501473:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762249/real 1789762249] req@ffff8dc00368d880 x1876699512932608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762249.501485:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512932608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762249.501490:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762249.501492:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762249.501494:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512932608:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762250.525089:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762251.549028:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762252.573184:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762253.597068:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762254.621093:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762254.621117:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762254.621139:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789762254.621140:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762254.621153:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2229 00000100:00080000:2.0:1789762254.621166:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762254.621173:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762254.621217:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512932736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762254.621238:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762254.621241:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762254.621242:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762254.621245:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2229 00000100:00080000:2.0:1789762254.621248:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762254.621251:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762254.621259:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762254.621270:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512932736:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762254.621271:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512932864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762254.621283:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762254.621285:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762254.621286:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762254.621289:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2229 00000100:00080000:2.0:1789762254.621292:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762254.621294:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762254.621297:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762254.621306:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512932992/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762254.621318:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762254.621320:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762254.621322:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762254.621325:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2229 00000100:00080000:2.0:1789762254.621327:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762254.621330:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789762254.621334:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:2.0:1789762254.621337:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762254.621347:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512933120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762254.621374:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762254/real 1789762254] req@ffff8dc00368d880 x1876699512932736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762254.621388:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512932736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762254.621396:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762254.621399:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762254.621409:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512932736:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762254.621428:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512932864:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762254.621475:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762254.621486:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762254/real 1789762254] req@ffff8dc00368fb80 x1876699512932864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762254.621496:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512932864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762254.621504:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762254.621507:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762254.621510:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512932864:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762254.621515:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512932992:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762254.621530:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762254.621537:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762254/real 1789762254] req@ffff8dc00368f480 x1876699512932992/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762254.621547:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512932992/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762254.621554:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762254.621556:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762254.621559:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512932992:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762254.621563:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512933120:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762254.621576:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762254.621583:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762254/real 1789762254] req@ffff8dc00368f100 x1876699512933120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762254.621591:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512933120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762254.621598:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762254.621600:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762254.621603:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512933120:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762255.645165:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762256.669135:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762257.693047:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762258.717157:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762259.741082:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1789762259.741085:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762259.741093:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762259.741098:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762259.741109:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2234 00000100:00080000:2.0:1789762259.741120:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762259.741127:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762259.741159:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512933248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762259.741181:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762259.741185:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762259.741187:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762259.741192:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2234 00000100:00080000:2.0:1789762259.741197:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762259.741203:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762259.741215:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762259.741224:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512933248:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762259.741232:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512933376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762259.741243:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762259.741245:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762259.741247:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762259.741251:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2234 00000100:00080000:2.0:1789762259.741254:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762259.741257:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762259.741262:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762259.741277:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512933504/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762259.741287:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762259.741290:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762259.741291:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762259.741294:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2234 00000100:00080000:2.0:1789762259.741298:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762259.741301:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762259.741305:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762259.741312:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512933632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762259.741319:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762259.741370:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762259/real 1789762259] req@ffff8dc00368ce00 x1876699512933248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762259.741386:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512933248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762259.741396:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762259.741400:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762259.741403:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512933248:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762259.741425:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512933376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762259.741449:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762259.741463:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762259/real 1789762259] req@ffff8dc00368ea00 x1876699512933376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762259.741478:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512933376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762259.741488:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762259.741490:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762259.741493:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512933376:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762259.741498:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512933504:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762259.741518:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762259.741527:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762259/real 1789762259] req@ffff8dc00368c000 x1876699512933504/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762259.741535:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512933504/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762259.741540:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762259.741543:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762259.741545:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512933504:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762259.741548:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512933632:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762259.741560:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762259.741566:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762259/real 1789762259] req@ffff8dc00368ca80 x1876699512933632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762259.741573:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512933632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762259.741578:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762259.741580:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762259.741582:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512933632:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762260.765042:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762261.789035:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762262.813017:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762263.837060:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762264.861014:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762264.861130:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762264.861143:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762264.861149:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762264.861163:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2239 00000100:00080000:2.0:1789762264.861194:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762264.861204:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762264.861271:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512933760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762264.861318:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762264.861322:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762264.861324:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762264.861329:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2239 00000100:00080000:2.0:1789762264.861333:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762264.861339:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762264.861370:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762264.861378:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512933760:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762264.861386:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512933888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762264.861401:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762264.861406:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762264.861408:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762264.861430:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2239 00000100:00080000:2.0:1789762264.861436:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762264.861441:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762264.861449:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762264.861463:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512934016/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762264.861480:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762264.861502:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762264.861506:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762264.861508:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762264.861516:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2239 00000100:00080000:2.0:1789762264.861519:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762264.861525:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762264.861531:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762264.861545:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762264/real 1789762264] req@ffff8dc00368df80 x1876699512933760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762264.861548:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512934144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762264.861656:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512933760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762264.861674:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762264.861679:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762264.861683:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512933760:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762264.861719:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512933888:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762264.861748:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762264.861765:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762264/real 1789762264] req@ffff8dc00368ed80 x1876699512933888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762264.861797:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512933888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762264.861808:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762264.861812:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762264.861816:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512933888:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762264.861825:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512934016:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762264.861852:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762264.861868:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762264/real 1789762264] req@ffff8dc00368e300 x1876699512934016/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762264.861904:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512934016/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762264.861954:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762264.861960:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762264.861965:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512934016:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762264.861974:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512934144:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762264.862000:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762264.862022:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762264/real 1789762264] req@ffff8dc00368f800 x1876699512934144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762264.862068:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512934144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762264.862075:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762264.862078:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762264.862080:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512934144:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762265.885100:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762266.909032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762267.933065:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762268.957051:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762269.981027:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762269.981063:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762269.981072:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762269.981077:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762269.981088:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2244 00000100:00080000:2.0:1789762269.981100:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762269.981107:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762269.981152:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512934272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762269.981181:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762269.981184:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762269.981186:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762269.981189:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2244 00000100:00080000:2.0:1789762269.981193:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762269.981196:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762269.981207:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762269.981218:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512934400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762269.981227:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762269.981233:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789762269.981235:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512934272:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762269.981236:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762269.981245:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2244 00000100:00080000:2.0:1789762269.981253:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762269.981258:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762269.981261:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762269.981272:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512934528/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762269.981280:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762269.981283:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762269.981285:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762269.981288:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2244 00000100:00080000:2.0:1789762269.981290:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762269.981293:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762269.981296:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762269.981306:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512934656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762269.981331:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762269.981400:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762269/real 1789762269] req@ffff8dc00368c700 x1876699512934272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762269.981430:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512934272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762269.981442:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762269.981447:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762269.981452:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512934272:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762269.981478:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512934400:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762269.981503:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762269.981516:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762269/real 1789762269] req@ffff8dc00368dc00 x1876699512934400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762269.981531:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512934400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762269.981540:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762269.981543:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762269.981547:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512934400:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762269.981555:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512934528:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762269.981578:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762269.981591:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762269/real 1789762269] req@ffff8dc00368e680 x1876699512934528/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762269.981604:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512934528/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762269.981620:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762269.981624:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762269.981628:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512934528:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762269.981635:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512934656:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762269.981657:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762269.981669:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762269/real 1789762269] req@ffff8dc00368d180 x1876699512934656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762269.981682:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512934656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762269.981691:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762269.981695:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762269.981699:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512934656:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762271.005000:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762272.029085:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762273.053027:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762274.077227:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762275.101118:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762275.101148:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762275.101154:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762275.101168:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2249 00000100:00080000:2.0:1789762275.101180:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762275.101190:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762275.101233:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512934784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762275.101312:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762275.101317:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762275.101320:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762275.101325:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2249 00000100:00080000:2.0:1789762275.101330:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762275.101335:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762275.101365:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762275.101385:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512934912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762275.101402:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762275.101411:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789762275.101413:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512934784:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762275.101414:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762275.101429:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2249 00000100:00080000:2.0:1789762275.101437:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1789762275.101442:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762275.101444:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762275.101456:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762275.101478:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512935040/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762275.101493:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762275.101498:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762275.101502:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789762275.101507:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762275.101514:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2249 00000100:00080000:2.0:1789762275.101523:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762275.101530:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762275.101556:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762275.101574:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762275/real 1789762275] req@ffff8dc00368d180 x1876699512934784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762275.101577:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512935168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762275.101605:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512934784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762275.101618:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762275.101622:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762275.101626:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512934784:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762275.101652:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512934912:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762275.101675:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762275.101688:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762275/real 1789762275] req@ffff8dc00368e680 x1876699512934912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762275.101701:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512934912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762275.101709:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762275.101713:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762275.101717:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512934912:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762275.101724:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512935040:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762275.101739:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762275.101759:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762275/real 1789762275] req@ffff8dc00368dc00 x1876699512935040/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762275.101776:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512935040/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762275.101785:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762275.101790:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762275.101795:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512935040:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762275.101801:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512935168:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762275.101819:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762275.101849:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762275/real 1789762275] req@ffff8dc00368c700 x1876699512935168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762275.101860:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512935168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762275.101868:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762275.101870:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762275.101872:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512935168:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762276.125020:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762277.149072:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762278.173108:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762279.197061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762280.221050:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1789762280.221061:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762280.221061:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762280.221065:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762280.221077:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2254 00000100:00080000:2.0:1789762280.221087:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762280.221094:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762280.221133:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512935296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762280.221157:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762280.221160:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762280.221161:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762280.221165:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2254 00000100:00080000:2.0:1789762280.221169:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762280.221172:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762280.221181:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762280.221185:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512935296:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762280.221195:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512935424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762280.221211:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762280.221214:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762280.221215:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762280.221218:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2254 00000100:00080000:2.0:1789762280.221221:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762280.221223:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762280.221227:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762280.221235:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512935552/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762280.221243:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762280.221245:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762280.221247:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762280.221249:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2254 00000100:00080000:2.0:1789762280.221251:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762280.221254:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762280.221257:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762280.221266:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512935680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762280.221274:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512935424:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762280.221300:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512935552:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762280.221316:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512935680:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762281.245158:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762282.269200:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762283.293315:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00010000:00010000:0.0:1789762283.293325:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762283.293362:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762283.293380:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762283.293435:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762283.293456:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762283.293472:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:0.0:1789762283.293482:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762280/real 1789762283] req@ffff8dc00368e300 x1876699512935424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762283.293526:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512935424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762283.293543:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762283.293550:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762283.293556:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512935424:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762283.293567:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762280/real 1789762283] req@ffff8dc00368ed80 x1876699512935552/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762283.293584:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512935552/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762283.293594:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762283.293598:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762283.293602:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512935552:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762283.293611:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762280/real 1789762283] req@ffff8dc00368f800 x1876699512935296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762283.293626:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512935296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762283.293634:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762283.293638:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762283.293642:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512935296:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762283.293672:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762280/real 1789762283] req@ffff8dc00368df80 x1876699512935680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762283.293686:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512935680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762283.293696:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762283.293700:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762283.293704:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512935680:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762284.317116:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762285.341031:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762285.341040:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762285.341045:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762285.341058:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2259 00000100:00080000:1.0:1789762285.341071:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762285.341080:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762285.341125:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512935808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1789762285.341129:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762285.341163:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762285.341169:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762285.341172:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762285.341178:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2259 00000100:00080000:1.0:1789762285.341184:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762285.341190:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762285.341203:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512935808:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1789762285.341203:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762285.341248:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512935936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762285.341264:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762285.341269:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762285.341273:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762285.341278:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2259 00000100:00080000:1.0:1789762285.341284:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762285.341293:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789762285.341296:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1789762285.341304:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762285.341323:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512936064/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762285.341362:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762285/real 1789762285] req@ffff8dc005c95180 x1876699512935808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762285.341381:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762285.341386:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762285.341388:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762285.341396:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2259 00000100:00080000:1.0:1789762285.341405:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762285.341414:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762285.341422:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762285.341426:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512935808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762285.341441:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512936192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762285.341444:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762285.341456:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762285.341467:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512935808:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762285.341512:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512935936:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762285.341545:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762285.341565:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762285/real 1789762285] req@ffff8dc005c97100 x1876699512935936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762285.341582:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512935936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762285.341593:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762285.341598:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762285.341603:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512935936:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762285.341611:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512936064:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762285.341638:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762285.341653:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762285/real 1789762285] req@ffff8dc005c94a80 x1876699512936064/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762285.341668:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512936064/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762285.341678:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762285.341682:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762285.341687:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512936064:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762285.341694:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512936192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762285.341720:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762285.341735:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762285/real 1789762285] req@ffff8dc005c95c00 x1876699512936192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762285.341748:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512936192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762285.341758:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762285.341762:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762285.341766:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512936192:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762286.365205:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762287.389073:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762288.413034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762289.437071:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762290.461040:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762290.461059:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762290.461069:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762290.461074:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762290.461084:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2265 00000100:00080000:2.0:1789762290.461094:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762290.461101:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762290.461143:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512936320/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762290.461176:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762290.461182:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762290.461186:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762290.461193:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2265 00000100:00080000:2.0:1789762290.461200:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762290.461206:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762290.461215:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512936320:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789762290.461216:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762290.461240:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512936448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762290.461254:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762290.461258:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762290.461259:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762290.461263:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2265 00000100:00080000:2.0:1789762290.461268:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762290.461274:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762290.461284:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762290.461303:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512936576/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762290.461312:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762290.461330:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762290.461335:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762290.461338:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762290.461343:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2265 00000100:00080000:2.0:1789762290.461348:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762290.461353:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762290.461358:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762290.461369:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762290/real 1789762290] req@ffff8dc00368ca80 x1876699512936320/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762290.461375:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512936704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762290.461432:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512936320/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762290.461448:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762290.461454:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762290.461459:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512936320:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762290.461488:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512936448:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762290.461509:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762290.461519:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762290/real 1789762290] req@ffff8dc00368c000 x1876699512936448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762290.461528:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512936448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762290.461534:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762290.461536:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762290.461538:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512936448:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762290.461543:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512936576:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762290.461555:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762290.461562:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762290/real 1789762290] req@ffff8dc00368ea00 x1876699512936576/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762290.461570:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512936576/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762290.461575:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762290.461577:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762290.461579:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512936576:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762290.461583:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512936704:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762290.461597:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762290.461608:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762290/real 1789762290] req@ffff8dc00368ce00 x1876699512936704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762290.461620:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512936704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762290.461629:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762290.461641:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762290.461645:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512936704:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762291.485045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762292.509007:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762293.533028:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762294.557045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762295.581176:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762295.581192:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762295.581200:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762295.581204:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762295.581214:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2270 00000100:00080000:2.0:1789762295.581224:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762295.581232:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762295.581269:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512936832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762295.581290:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762295.581293:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762295.581295:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762295.581298:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2270 00000100:00080000:2.0:1789762295.581300:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762295.581303:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762295.581311:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762295.581319:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512936960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762295.581326:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762295.581330:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762295.581331:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762295.581334:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2270 00000100:00080000:2.0:1789762295.581336:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762295.581341:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762295.581344:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762295.581352:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512937088/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762295.581359:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762295.581362:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762295.581364:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762295.581367:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2270 00000100:00080000:2.0:1789762295.581369:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762295.581372:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762295.581375:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762295.581386:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512937216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762295.581392:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512936832:192.168.203.156@tcp:250:kworker.0 00000800:00000100:0.0:1789762295.581524:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762295.581597:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762295/real 1789762295] req@ffff8dc00368f100 x1876699512936832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762295.581636:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512936832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762295.581660:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762295.581669:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762295.581677:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512936832:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762295.581693:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512936960:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762295.581742:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762295.581770:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762295/real 1789762295] req@ffff8dc00368f480 x1876699512936960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762295.581822:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512936960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762295.581841:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762295.581849:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762295.581856:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512936960:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762295.581870:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512937088:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762295.581956:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762295.581982:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762295/real 1789762295] req@ffff8dc00368fb80 x1876699512937088/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762295.582010:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512937088/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762295.582041:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762295.582047:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762295.582054:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512937088:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762295.582103:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512937216:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762295.582149:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762295.582176:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762295/real 1789762295] req@ffff8dc00368d880 x1876699512937216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762295.582202:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512937216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762295.582219:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762295.582226:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762295.582233:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512937216:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762296.605020:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762297.629107:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762298.653013:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762299.677057:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762300.700989:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762300.700999:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762300.701003:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762300.701016:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2275 00000100:00080000:2.0:1789762300.701026:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762300.701034:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789762300.701038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762300.701073:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512937344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762300.701097:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762300.701100:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762300.701101:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762300.701105:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2275 00000100:00080000:2.0:1789762300.701108:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762300.701112:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762300.701120:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762300.701130:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512937344:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762300.701134:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512937472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762300.701144:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762300.701146:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762300.701147:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762300.701151:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2275 00000100:00080000:2.0:1789762300.701153:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762300.701155:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762300.701159:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762300.701168:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512937600/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762300.701175:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762300.701178:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762300.701179:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762300.701182:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2275 00000100:00080000:2.0:1789762300.701187:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762300.701189:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762300.701192:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762300.701202:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762300.701206:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512937728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762300.701252:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762300/real 1789762300] req@ffff8dc00368d880 x1876699512937344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762300.701272:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512937344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762300.701285:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762300.701288:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762300.701291:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512937344:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762300.701311:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512937472:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762300.701328:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762300.701338:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762300/real 1789762300] req@ffff8dc00368fb80 x1876699512937472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762300.701346:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512937472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762300.701352:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762300.701354:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762300.701356:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512937472:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762300.701360:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512937600:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762300.701372:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762300.701378:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762300/real 1789762300] req@ffff8dc00368f480 x1876699512937600/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762300.701386:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512937600/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762300.701392:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762300.701393:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762300.701395:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512937600:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762300.701399:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512937728:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762300.701410:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762300.701416:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762300/real 1789762300] req@ffff8dc00368f100 x1876699512937728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762300.701424:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512937728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762300.701430:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762300.701431:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762300.701433:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512937728:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762301.725012:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762302.749004:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762303.773038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762304.797000:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762305.821027:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762305.821044:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762305.821053:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762305.821057:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762305.821068:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2280 00000100:00080000:1.0:1789762305.821079:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762305.821085:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762305.821124:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512937856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762305.821157:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762305.821160:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762305.821163:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762305.821167:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2280 00000100:00080000:1.0:1789762305.821170:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762305.821174:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762305.821184:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762305.821195:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512937984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762305.821205:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762305.821209:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762305.821211:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762305.821214:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2280 00000100:00080000:1.0:1789762305.821217:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762305.821223:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512937856:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789762305.821225:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762305.821236:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762305.821250:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512938112/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762305.821260:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762305.821265:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762305.821267:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762305.821271:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2280 00000100:00080000:1.0:1789762305.821284:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762305.821289:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762305.821293:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762305.821301:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762305.821316:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512938240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762305.821358:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762305/real 1789762305] req@ffff8dc005c97800 x1876699512937856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762305.821379:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512937856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762305.821389:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762305.821393:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762305.821397:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512937856:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762305.821406:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512937984:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762305.821427:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762305.821438:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762305/real 1789762305] req@ffff8dc005c96a00 x1876699512937984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762305.821449:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512937984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762305.821457:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762305.821460:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762305.821463:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512937984:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762305.821487:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512938112:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762305.821502:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762305.821511:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762305/real 1789762305] req@ffff8dc005c95f80 x1876699512938112/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762305.821522:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512938112/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762305.821533:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762305.821538:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762305.821543:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512938112:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762305.821551:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512938240:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762305.821580:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762305.821597:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762305/real 1789762305] req@ffff8dc005c95500 x1876699512938240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762305.821617:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512938240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762305.821630:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762305.821635:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762305.821640:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512938240:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762306.845027:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762307.869078:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762308.893061:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762309.917158:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762310.941018:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762310.941029:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762310.941033:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762310.941045:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2285 00000100:00080000:1.0:1789762310.941056:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762310.941065:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762310.941125:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512938368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762310.941156:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1789762310.941159:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762310.941163:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762310.941169:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762310.941176:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2285 00000100:00080000:1.0:1789762310.941182:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762310.941189:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762310.941202:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762310.941204:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512938368:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762310.941226:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512938496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762310.941241:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762310.941245:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762310.941247:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762310.941253:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2285 00000100:00080000:1.0:1789762310.941273:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762310.941281:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762310.941290:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789762310.941292:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762310.941316:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512938624/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762310.941349:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762310.941353:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762310.941355:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762310.941360:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2285 00000100:00080000:1.0:1789762310.941365:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789762310.941374:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762310/real 1789762310] req@ffff8dc005c94700 x1876699512938368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762310.941375:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762310.941387:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762310.941404:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512938752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762310.941427:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512938368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762310.941460:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762310.941465:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762310.941470:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512938368:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762310.941495:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512938496:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762310.941541:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762310.941558:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762310/real 1789762310] req@ffff8dc005c94380 x1876699512938496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762310.941578:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512938496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762310.941587:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762310.941590:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762310.941593:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512938496:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762310.941600:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512938624:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762310.941636:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762310.941651:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762310/real 1789762310] req@ffff8dc005c97b80 x1876699512938624/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762310.941663:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512938624/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762310.941673:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762310.941677:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762310.941680:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512938624:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762310.941707:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512938752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762310.941734:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762310.941749:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762310/real 1789762310] req@ffff8dc005c96300 x1876699512938752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762310.941760:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512938752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762310.941773:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762310.941776:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762310.941778:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512938752:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762311.965040:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762312.989190:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762314.013133:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762315.037071:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762316.060998:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762316.061009:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762316.061013:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762316.061022:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2290 00000100:00080000:3.0:1789762316.061031:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1789762316.061036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762316.061038:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762316.061073:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699512938880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762316.061094:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762316.061097:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762316.061099:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762316.061102:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2290 00000100:00080000:3.0:1789762316.061105:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762316.061108:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762316.061116:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762316.061125:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512938880:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762316.061127:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699512939008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762316.061139:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762316.061143:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762316.061145:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762316.061149:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2290 00000100:00080000:3.0:1789762316.061153:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762316.061158:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762316.061165:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762316.061180:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512939136/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762316.061186:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762316.061199:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762316.061203:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762316.061204:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762316.061209:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2290 00000100:00080000:3.0:1789762316.061212:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762316.061215:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762316.061220:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762316.061232:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762316/real 1789762316] req@ffff8dc0107afb80 x1876699512938880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762316.061233:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512939264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762316.061274:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699512938880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762316.061283:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762316.061286:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762316.061289:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512938880:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762316.061310:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512939008:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762316.061334:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762316.061348:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762316/real 1789762316] req@ffff8dc0107adc00 x1876699512939008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762316.061362:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699512939008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762316.061379:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762316.061383:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762316.061387:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512939008:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762316.061394:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512939136:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762316.061418:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762316.061431:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762316/real 1789762316] req@ffff8dc0107aca80 x1876699512939136/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762316.061445:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512939136/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762316.061453:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762316.061457:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762316.061461:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512939136:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762316.061468:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512939264:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762316.061490:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762316.061502:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762316/real 1789762316] req@ffff8dc0107ae300 x1876699512939264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762316.061515:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512939264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762316.061524:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762316.061528:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762316.061532:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512939264:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762317.085036:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762318.109056:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762319.133201:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762320.157199:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762321.181028:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762321.181044:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762321.181050:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762321.181065:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2295 00000100:00080000:3.0:1789762321.181078:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1789762321.181086:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762321.181097:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762321.181151:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512939392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762321.181182:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762321.181186:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762321.181188:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762321.181192:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2295 00000100:00080000:3.0:1789762321.181197:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762321.181201:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762321.181213:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762321.181231:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512939520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762321.181232:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512939392:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762321.181259:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762321.181264:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762321.181267:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762321.181273:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2295 00000100:00080000:3.0:1789762321.181278:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762321.181283:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762321.181291:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762321.181304:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512939648/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762321.181315:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762321.181316:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762321.181325:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762321.181327:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762321.181331:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2295 00000100:00080000:3.0:1789762321.181335:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762321.181338:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762321.181343:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762321.181352:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512939776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762321.181377:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762321/real 1789762321] req@ffff8dc0107ac700 x1876699512939392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762321.181393:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512939392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762321.181402:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762321.181406:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762321.181409:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512939392:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762321.181429:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512939520:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762321.181448:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762321.181457:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762321/real 1789762321] req@ffff8dc0107ac000 x1876699512939520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762321.181466:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512939520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762321.181472:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762321.181475:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762321.181477:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512939520:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762321.181481:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512939648:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762321.181493:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762321.181500:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762321/real 1789762321] req@ffff8dc0107ad500 x1876699512939648/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762321.181508:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512939648/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762321.181514:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762321.181515:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762321.181517:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512939648:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762321.181521:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512939776:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762321.181532:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762321.181539:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762321/real 1789762321] req@ffff8dc0107aed80 x1876699512939776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762321.181547:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512939776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762321.181552:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762321.181554:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762321.181556:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512939776:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762322.205081:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762323.229017:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762324.253030:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762325.277157:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762326.301004:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762326.301016:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762326.301022:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762326.301033:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2300 00010000:00010000:2.0:1789762326.301040:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762326.301043:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762326.301052:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762326.301090:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512939904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762326.301118:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762326.301123:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762326.301125:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762326.301130:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2300 00000100:00080000:1.0:1789762326.301134:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762326.301148:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762326.301158:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762326.301174:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512940032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762326.301175:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512939904:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789762326.301193:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762326.301198:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762326.301200:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762326.301206:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2300 00000100:00080000:1.0:1789762326.301210:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762326.301215:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762326.301221:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762326.301234:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512940160/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762326.301246:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789762326.301247:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762326.301253:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762326.301255:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762326.301262:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2300 00000100:00080000:1.0:1789762326.301266:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762326.301271:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762326.301278:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762326.301290:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512940288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762326.301305:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762326/real 1789762326] req@ffff8dc005c94e00 x1876699512939904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762326.301328:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512939904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762326.301342:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762326.301346:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762326.301351:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512939904:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762326.301375:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512940032:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762326.301402:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762326.301417:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762326/real 1789762326] req@ffff8dc005c97480 x1876699512940032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762326.301431:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512940032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762326.301452:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762326.301456:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762326.301460:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512940032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762326.301467:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512940160:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762326.301490:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762326.301502:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762326/real 1789762326] req@ffff8dc005c96680 x1876699512940160/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762326.301514:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512940160/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762326.301523:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762326.301526:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762326.301530:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512940160:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762326.301536:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512940288:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762326.301562:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762326.301590:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762326/real 1789762326] req@ffff8dc005c95880 x1876699512940288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762326.301605:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512940288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762326.301614:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762326.301626:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762326.301630:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512940288:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762327.325016:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762328.349004:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762329.373049:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762330.397098:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762331.421023:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762331.421034:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762331.421039:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762331.421052:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2306 00000100:00080000:1.0:1789762331.421065:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762331.421075:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789762331.421079:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789762331.421120:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96d80 x1876699512940416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762331.421141:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762331.421144:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762331.421146:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762331.421148:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2306 00000100:00080000:1.0:1789762331.421151:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762331.421153:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762331.421162:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762331.421175:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512940544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762331.421183:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762331.421185:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762331.421187:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762331.421193:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2306 00000100:00100000:0.0:1789762331.421195:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512940416:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789762331.421195:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762331.421201:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762331.421207:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762331.421216:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512940672/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762331.421222:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762331.421225:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762331.421226:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762331.421229:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2306 00000100:00080000:1.0:1789762331.421231:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762331.421234:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762331.421237:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762331.421244:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512940800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762331.421269:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762331.421308:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762331/real 1789762331] req@ffff8dc005c96d80 x1876699512940416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762331.421322:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96d80 x1876699512940416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762331.421336:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762331.421340:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762331.421343:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512940416:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762331.421363:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512940544:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762331.421379:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762331.421388:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762331/real 1789762331] req@ffff8dc005c95880 x1876699512940544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762331.421396:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512940544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762331.421402:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762331.421404:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762331.421406:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512940544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762331.421409:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512940672:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762331.421421:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762331.421433:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762331/real 1789762331] req@ffff8dc005c96680 x1876699512940672/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762331.421441:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512940672/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762331.421446:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762331.421448:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762331.421450:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512940672:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762331.421453:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512940800:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762331.421465:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762331.421471:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762331/real 1789762331] req@ffff8dc005c97480 x1876699512940800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762331.421478:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512940800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762331.421488:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762331.421489:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762331.421492:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512940800:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762332.445254:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762333.469084:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762334.493091:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762335.517127:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762336.541020:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762336.541028:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762336.541036:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762336.541041:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762336.541052:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2311 00000100:00080000:3.0:1789762336.541062:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762336.541069:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762336.541109:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512940928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762336.541137:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762336.541141:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762336.541143:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762336.541148:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2311 00000100:00080000:3.0:1789762336.541153:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762336.541159:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762336.541186:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762336.541197:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512940928:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762336.541207:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512941056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762336.541219:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762336.541222:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762336.541224:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762336.541228:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2311 00000100:00080000:3.0:1789762336.541232:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762336.541236:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762336.541240:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762336.541263:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512941184/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762336.541274:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762336.541277:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762336.541279:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762336.541282:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2311 00000100:00080000:3.0:1789762336.541286:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762336.541292:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762336.541297:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762336.541303:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762336.541315:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512941312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762336.541343:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762336/real 1789762336] req@ffff8dc0107ad180 x1876699512940928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762336.541363:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512940928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762336.541373:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762336.541377:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762336.541380:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512940928:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762336.541404:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512941056:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762336.541423:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762336.541434:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762336/real 1789762336] req@ffff8dc0107af480 x1876699512941056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762336.541445:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512941056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762336.541453:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762336.541455:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762336.541458:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512941056:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762336.541464:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512941184:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762336.541479:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762336.541487:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762336/real 1789762336] req@ffff8dc0107adf80 x1876699512941184/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762336.541538:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512941184/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762336.541544:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762336.541547:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762336.541549:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512941184:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762336.541553:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512941312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762336.541568:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762336.541576:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762336/real 1789762336] req@ffff8dc0107ac380 x1876699512941312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762336.541607:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512941312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762336.541614:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762336.541617:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762336.541619:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512941312:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762337.565091:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762338.589088:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762339.613027:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762340.637052:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762341.660992:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762341.661001:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762341.661004:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762341.661013:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2316 00000100:00080000:2.0:1789762341.661022:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762341.661027:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789762341.661049:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762341.661060:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512941440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762341.661082:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762341.661084:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762341.661086:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762341.661089:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2316 00000100:00080000:2.0:1789762341.661091:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762341.661095:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762341.661102:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762341.661116:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512941440:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762341.661116:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512941568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762341.661130:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762341.661145:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762341.661146:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762341.661149:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2316 00000100:00080000:2.0:1789762341.661151:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762341.661154:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762341.661157:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762341.661168:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512941696/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762341.661176:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762341.661178:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762341.661179:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762341.661182:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2316 00000100:00080000:2.0:1789762341.661185:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762341.661187:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762341.661190:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762341.661195:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762341.661199:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512941824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762341.661252:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762341/real 1789762341] req@ffff8dc00368ce00 x1876699512941440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762341.661266:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512941440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762341.661274:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762341.661277:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762341.661280:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512941440:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762341.661310:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512941568:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762341.661326:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762341.661334:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762341/real 1789762341] req@ffff8dc00368ea00 x1876699512941568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762341.661343:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512941568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762341.661349:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762341.661351:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762341.661353:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512941568:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762341.661357:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512941696:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762341.661369:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762341.661375:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762341/real 1789762341] req@ffff8dc00368c000 x1876699512941696/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762341.661383:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512941696/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762341.661393:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762341.661395:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762341.661398:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512941696:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762341.661401:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512941824:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762341.661411:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762341.661417:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762341/real 1789762341] req@ffff8dc00368ca80 x1876699512941824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762341.661424:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512941824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762341.661430:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762341.661431:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762341.661433:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512941824:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762342.685025:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762343.709090:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762344.733148:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762345.757110:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762346.781065:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762346.781084:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762346.781091:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762346.781108:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2321 00000100:00080000:2.0:1789762346.781125:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762346.781167:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789762346.781216:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762346.781241:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512941952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762346.781305:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762346.781311:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762346.781314:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762346.781320:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2321 00000100:00080000:2.0:1789762346.781328:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762346.781337:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762346.781351:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762346.781373:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512942080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762346.781403:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512941952:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762346.781411:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762346.781444:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762346.781448:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762346.781457:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2321 00000100:00080000:2.0:1789762346.781464:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762346.781470:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762346.781478:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762346.781494:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512942208/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762346.781507:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762346.781511:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762346.781513:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762346.781517:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2321 00000100:00080000:2.0:1789762346.781529:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762346.781532:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762346.781536:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762346.781545:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512942336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762346.781648:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512942080:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762346.782087:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512942208:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789762346.782155:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512942336:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762347.805107:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762348.829157:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762349.853059:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789762349.853451:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:2.0:1789762349.853511:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789762349.853542:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789762349.853615:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789762349.853632:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789762349.853646:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:3.0:1789762349.853699:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762346/real 1789762349] req@ffff8dc00368f800 x1876699512942080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762349.853749:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512942080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762349.853774:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762349.853782:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762349.853791:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512942080:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762349.853806:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762346/real 1789762349] req@ffff8dc00368ed80 x1876699512942208/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762349.853830:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512942208/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762349.853845:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762349.853852:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762349.853858:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512942208:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789762349.853870:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762346/real 1789762349] req@ffff8dc00368e300 x1876699512942336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762349.853891:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512942336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762349.853961:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762349.853968:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762349.853975:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512942336:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762349.853989:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762346/real 1789762349] req@ffff8dc00368df80 x1876699512941952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762371 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762349.854015:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512941952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762371 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762349.854029:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762349.854035:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762349.854042:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512941952:192.168.203.156@tcp:250:kworker.0 00010000:00010000:3.0:1789762350.877050:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762351.901066:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762351.901098:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762351.901110:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762351.901116:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762351.901130:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2326 00000100:00080000:2.0:1789762351.901142:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762351.901151:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762351.901214:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512942464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762351.901265:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762351.901268:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762351.901270:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762351.901273:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2326 00000100:00080000:2.0:1789762351.901275:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762351.901278:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762351.901288:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762351.901298:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512942592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762351.901327:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762351.901331:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762351.901333:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762351.901338:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2326 00000100:00080000:2.0:1789762351.901342:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762351.901348:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762351.901352:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512942464:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789762351.901355:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762351.901372:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512942720/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762351.901401:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762351.901406:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762351.901408:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762351.901412:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2326 00000100:00080000:2.0:1789762351.901417:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762351.901422:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762351.901428:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762351.901440:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512942848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762351.901463:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762351.901520:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762351/real 1789762351] req@ffff8dc00368c700 x1876699512942464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762351.901542:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512942464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762351.901556:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762351.901561:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762351.901566:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512942464:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762351.901574:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512942592:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762351.901599:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762351.901613:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762351/real 1789762351] req@ffff8dc00368dc00 x1876699512942592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762351.901625:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512942592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762351.901639:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762351.901642:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762351.901646:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512942592:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762351.901677:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512942720:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789762351.901699:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762351.901710:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762351/real 1789762351] req@ffff8dc00368e680 x1876699512942720/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762351.901720:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512942720/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762351.901728:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762351.901730:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762351.901734:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512942720:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789762351.901740:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512942848:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762351.901758:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762351.901768:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762351/real 1789762351] req@ffff8dc00368d180 x1876699512942848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762351.901780:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512942848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762351.901788:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762351.902146:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762351.902152:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512942848:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762352.925015:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762353.949159:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762354.973114:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762355.997022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762357.021043:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762357.021053:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762357.021058:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762357.021072:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2331 00000100:00080000:2.0:1789762357.021087:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762357.021098:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789762357.021121:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762357.021164:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512942976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762357.021194:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762357.021214:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762357.021216:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762357.021221:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2331 00000100:00080000:2.0:1789762357.021227:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762357.021234:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762357.021247:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762357.021259:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512942976:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762357.021288:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512943104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762357.021315:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762357.021320:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762357.021323:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762357.021329:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2331 00000100:00080000:2.0:1789762357.021336:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762357.021360:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762357.021386:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762357.021402:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512943232/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762357.021418:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762357.021419:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762357.021428:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762357.021430:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762357.021435:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2331 00000100:00080000:2.0:1789762357.021441:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762357.021446:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762357.021451:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762357.021475:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512943360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762357.021501:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762357/real 1789762357] req@ffff8dc00368d180 x1876699512942976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762357.021533:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512942976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762357.021553:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762357.021559:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762357.021566:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512942976:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762357.021602:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512943104:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762357.021640:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762357.021701:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762357/real 1789762357] req@ffff8dc00368e680 x1876699512943104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762357.021720:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512943104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762357.021743:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762357.021748:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762357.021752:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512943104:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762357.021765:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512943232:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789762357.021792:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762357.021817:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762357/real 1789762357] req@ffff8dc00368dc00 x1876699512943232/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762357.021835:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512943232/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762357.021858:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762357.021864:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762357.021880:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512943232:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789762357.021889:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512943360:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762357.021971:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762357.022015:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762357/real 1789762357] req@ffff8dc00368c700 x1876699512943360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762357.022034:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512943360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762357.022049:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762357.022054:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762357.022059:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512943360:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762358.045105:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762359.069076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762360.093114:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762361.117086:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762362.141024:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762362.141035:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762362.141040:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762362.141051:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2336 00000100:00080000:3.0:1789762362.141062:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762362.141071:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789762362.141082:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762362.141114:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512943488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762362.141147:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762362.141150:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762362.141152:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762362.141155:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2336 00000100:00080000:3.0:1789762362.141159:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762362.141163:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762362.141171:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762362.141182:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512943488:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762362.141183:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512943616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762362.141198:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762362.141200:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762362.141202:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762362.141205:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2336 00000100:00080000:3.0:1789762362.141208:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762362.141210:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762362.141219:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762362.141228:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512943744/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762362.141244:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789762362.141251:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762362.141251:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762362.141257:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762362.141263:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2336 00000100:00080000:3.0:1789762362.141269:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762362.141275:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762362.141281:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762362.141298:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512943872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762362.141315:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762362/real 1789762362] req@ffff8dc0107af800 x1876699512943488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762362.141336:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512943488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762362.141345:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762362.141348:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762362.141351:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512943488:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762362.141383:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512943616:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762362.141400:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762362.141409:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762362/real 1789762362] req@ffff8dc0107ae680 x1876699512943616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762362.141418:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512943616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762362.141424:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762362.141426:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762362.141428:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512943616:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762362.141432:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512943744:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762362.141443:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762362.141449:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762362/real 1789762362] req@ffff8dc0107ace00 x1876699512943744/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762362.141457:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512943744/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762362.141462:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762362.141464:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762362.141466:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512943744:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762362.141469:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512943872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762362.141480:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762362.141485:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762362/real 1789762362] req@ffff8dc0107af100 x1876699512943872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762362.141493:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512943872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762362.141498:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762362.141500:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762362.141502:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512943872:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762363.165029:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762364.189076:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762365.213029:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762366.237112:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762367.260970:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762367.260978:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762367.260982:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762367.260991:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2341 00000100:00080000:3.0:1789762367.260999:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762367.261006:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789762367.261039:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762367.261040:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512944000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762367.261062:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762367.261065:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762367.261067:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762367.261069:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2341 00000100:00080000:3.0:1789762367.261073:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762367.261076:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762367.261085:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762367.261094:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512944128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762367.261103:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512944000:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762367.261104:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762367.261111:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762367.261113:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762367.261117:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2341 00000100:00080000:3.0:1789762367.261119:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762367.261121:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762367.261125:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762367.261134:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512944256/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762367.261141:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762367.261145:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762367.261146:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762367.261149:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2341 00000100:00080000:3.0:1789762367.261151:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762367.261153:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762367.261157:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762367.261163:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512944384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762367.261168:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762367.261207:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762367/real 1789762367] req@ffff8dc0107ad880 x1876699512944000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762367.261220:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512944000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762367.261227:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762367.261230:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762367.261233:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512944000:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762367.261252:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512944128:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762367.261283:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762367.261294:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762367/real 1789762367] req@ffff8dc0107aea00 x1876699512944128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762367.261303:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512944128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762367.261331:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762367.261334:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762367.261336:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512944128:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762367.261341:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512944256:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762367.261355:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762367.261363:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762367/real 1789762367] req@ffff8dc0107af100 x1876699512944256/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762367.261371:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512944256/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762367.261387:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762367.261390:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762367.261392:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512944256:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762367.261395:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512944384:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762367.261405:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762367.261410:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762367/real 1789762367] req@ffff8dc0107ace00 x1876699512944384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762367.261418:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512944384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762367.261423:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762367.261425:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762367.261428:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512944384:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762368.285215:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762369.309079:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762370.333067:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762371.356999:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762372.380999:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762372.381010:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762372.381015:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762372.381027:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2346 00010000:00010000:1.0:1789762372.381028:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762372.381038:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762372.381047:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762372.381083:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512944512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762372.381104:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762372.381107:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762372.381109:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762372.381112:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2346 00000100:00080000:3.0:1789762372.381115:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762372.381118:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762372.381126:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762372.381134:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512944640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762372.381143:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762372.381147:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789762372.381148:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512944512:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762372.381149:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762372.381157:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2346 00000100:00080000:3.0:1789762372.381171:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762372.381174:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762372.381177:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762372.381186:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512944768/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762372.381194:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762372.381197:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762372.381199:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762372.381201:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2346 00000100:00080000:3.0:1789762372.381203:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762372.381206:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762372.381209:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762372.381215:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512944896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762372.381246:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762372.381302:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762372/real 1789762372] req@ffff8dc0107ae680 x1876699512944512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762372.381321:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512944512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762372.381341:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762372.381346:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762372.381350:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512944512:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762372.381374:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512944640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762372.381399:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762372.381414:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762372/real 1789762372] req@ffff8dc0107af800 x1876699512944640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762372.381427:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512944640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762372.381437:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762372.381440:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762372.381444:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512944640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762372.381451:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512944768:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762372.381471:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762372.381484:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762372/real 1789762372] req@ffff8dc0107ac380 x1876699512944768/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762372.381505:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512944768/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762372.381514:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762372.381517:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762372.381521:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512944768:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762372.381527:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512944896:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762372.381548:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762372.381571:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762372/real 1789762372] req@ffff8dc0107adf80 x1876699512944896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762372.381583:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512944896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762372.381592:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762372.381595:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762372.381599:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512944896:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762373.405136:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762374.429036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762375.453140:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762376.477180:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762377.501097:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762377.501106:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762377.501110:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762377.501120:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2352 00000100:00080000:2.0:1789762377.501131:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762377.501139:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789762377.501168:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762377.501178:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512945024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762377.501222:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762377.501226:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762377.501229:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762377.501234:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2352 00000100:00080000:2.0:1789762377.501239:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762377.501243:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762377.501251:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762377.501265:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512945152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762377.501273:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512945024:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762377.501279:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762377.501290:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762377.501292:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762377.501296:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2352 00000100:00080000:2.0:1789762377.501299:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762377.501301:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762377.501306:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762377.501317:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512945280/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762377.501326:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762377.501329:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762377.501330:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762377.501334:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2352 00000100:00080000:2.0:1789762377.501337:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762377.501340:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762377.501346:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762377.501355:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512945408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762377.501402:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762377.501472:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762377/real 1789762377] req@ffff8dc00368df80 x1876699512945024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762377.501504:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512945024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762377.501524:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762377.501531:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762377.501537:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512945024:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762377.501570:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512945152:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762377.501600:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762377.501614:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762377/real 1789762377] req@ffff8dc00368e300 x1876699512945152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762377.501627:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512945152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762377.501636:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762377.501640:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762377.501643:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512945152:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762377.501650:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512945280:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762377.501686:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762377.501695:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762377/real 1789762377] req@ffff8dc00368ed80 x1876699512945280/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762377.501706:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512945280/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762377.501713:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762377.501716:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762377.501719:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512945280:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762377.501725:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512945408:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762377.501741:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762377.501749:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762377/real 1789762377] req@ffff8dc00368f800 x1876699512945408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762377.501779:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512945408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762377.501788:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762377.501792:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762377.501796:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512945408:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762378.525057:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762379.549050:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762380.573035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762381.597069:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762382.621035:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762382.621049:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762382.621056:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762382.621070:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2357 00000100:00080000:2.0:1789762382.621110:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762382.621122:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789762382.621140:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762382.621184:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512945536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762382.621219:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762382.621224:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762382.621227:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762382.621233:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2357 00000100:00080000:2.0:1789762382.621239:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762382.621245:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762382.621267:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762382.621286:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512945536:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762382.621289:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512945664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762382.621330:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762382.621335:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762382.621338:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762382.621346:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2357 00000100:00080000:2.0:1789762382.621352:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762382.621358:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762382.621366:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762382.621383:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512945792/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762382.621396:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762382.621400:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762382.621402:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762382.621406:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2357 00000100:00080000:2.0:1789762382.621411:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789762382.621413:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762382.621425:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762382.621434:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762382.621462:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512945920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762382.621475:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762382/real 1789762382] req@ffff8dc00368ca80 x1876699512945536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762382.621500:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512945536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762382.621512:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762382.621517:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762382.621520:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512945536:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762382.621553:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512945664:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762382.621580:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762382.621596:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762382/real 1789762382] req@ffff8dc00368c000 x1876699512945664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762382.621607:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512945664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762382.621614:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762382.621618:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762382.621620:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512945664:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762382.621626:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512945792:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762382.621667:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762382.621684:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762382/real 1789762382] req@ffff8dc00368ea00 x1876699512945792/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762382.621702:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512945792/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762382.621716:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762382.621734:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762382.621749:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512945792:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762382.621759:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512945920:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762382.621803:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762382.621842:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762382/real 1789762382] req@ffff8dc00368ce00 x1876699512945920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762382.621856:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512945920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762382.621864:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762382.621868:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762382.621874:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512945920:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762383.645082:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762384.669086:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762385.693077:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762386.717033:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762387.741045:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762387.741073:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762387.741083:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762387.741089:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762387.741102:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2362 00000100:00080000:1.0:1789762387.741127:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762387.741135:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762387.741177:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512946048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762387.741203:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762387.741205:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762387.741207:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762387.741210:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2362 00000100:00080000:1.0:1789762387.741213:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762387.741216:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762387.741225:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762387.741238:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512946048:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762387.741241:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512946176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762387.741256:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762387.741259:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762387.741262:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762387.741267:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2362 00000100:00080000:1.0:1789762387.741271:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762387.741276:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762387.741283:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762387.741297:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512946304/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762387.741307:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762387.741309:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1789762387.741313:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789762387.741313:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762387.741321:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2362 00000100:00080000:1.0:1789762387.741326:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762387.741330:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762387.741334:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762387.741343:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512946432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762387.741363:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762387/real 1789762387] req@ffff8dc005c94e00 x1876699512946048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762387.741379:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512946048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762387.741388:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762387.741391:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762387.741394:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512946048:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762387.741415:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512946176:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762387.741431:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762387.741440:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762387/real 1789762387] req@ffff8dc005c96300 x1876699512946176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762387.741447:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512946176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762387.741453:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762387.741455:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762387.741457:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512946176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762387.741461:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512946304:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762387.741472:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762387.741478:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762387/real 1789762387] req@ffff8dc005c97b80 x1876699512946304/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762387.741485:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512946304/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762387.741490:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762387.741492:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762387.741494:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512946304:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762387.741498:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512946432:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762387.741517:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762387.741528:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762387/real 1789762387] req@ffff8dc005c94380 x1876699512946432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762412 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762387.741549:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512946432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762412 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762387.741558:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762387.741561:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762387.741564:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512946432:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762388.765081:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762389.789032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762390.813068:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762391.837069:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762392.861007:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762392.861018:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762392.861023:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762392.861034:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2367 00010000:00010000:3.0:1789762392.861041:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762392.861045:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762392.861052:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762392.861098:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512946560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762392.861124:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762392.861126:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762392.861128:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762392.861131:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2367 00000100:00080000:2.0:1789762392.861134:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762392.861136:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762392.861149:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762392.861162:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512946688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762392.861167:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512946560:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762392.861178:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762392.861182:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762392.861183:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762392.861186:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2367 00000100:00080000:2.0:1789762392.861190:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762392.861192:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762392.861196:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762392.861205:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512946816/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762392.861213:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762392.861215:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762392.861216:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762392.861219:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2367 00000100:00080000:2.0:1789762392.861224:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762392.861227:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762392.861230:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762392.861233:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762392.861244:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512946944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762392.861276:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762392/real 1789762392] req@ffff8dc00368f100 x1876699512946560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762392.861293:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512946560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762392.861302:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762392.861305:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762392.861308:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512946560:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762392.861328:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512946688:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762392.861343:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762392.861352:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762392/real 1789762392] req@ffff8dc00368f480 x1876699512946688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762392.861360:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512946688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762392.861370:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762392.861372:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762392.861374:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512946688:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762392.861378:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512946816:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762392.861388:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762392.861394:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762392/real 1789762392] req@ffff8dc00368fb80 x1876699512946816/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762392.861402:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512946816/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762392.861407:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762392.861408:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762392.861410:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512946816:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762392.861414:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512946944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762392.861424:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762392.861429:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762392/real 1789762392] req@ffff8dc00368d880 x1876699512946944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762392.861436:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512946944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762392.861441:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762392.861443:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762392.861446:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512946944:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762393.885196:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762394.909038:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762395.933045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762396.957124:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762397.981020:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762397.981041:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762397.981047:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762397.981060:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2372 00010000:00010000:1.0:1789762397.981073:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762397.981074:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762397.981091:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762397.981135:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512947072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762397.981158:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762397.981161:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762397.981162:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762397.981165:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2372 00000100:00080000:3.0:1789762397.981167:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762397.981170:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762397.981179:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762397.981189:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512947200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762397.981196:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762397.981202:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512947072:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762397.981202:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762397.981205:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762397.981213:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2372 00000100:00080000:3.0:1789762397.981215:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762397.981218:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762397.981221:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762397.981230:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512947328/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762397.981238:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762397.981241:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762397.981242:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762397.981245:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2372 00000100:00080000:3.0:1789762397.981247:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762397.981250:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762397.981253:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762397.981259:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512947456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762397.981278:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762397.981318:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762397/real 1789762397] req@ffff8dc0107af480 x1876699512947072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762397.981338:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512947072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762397.981350:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762397.981354:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762397.981359:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512947072:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762397.981386:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512947200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762397.981405:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762397.981415:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762397/real 1789762397] req@ffff8dc0107ad180 x1876699512947200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762397.981424:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512947200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762397.981430:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762397.981433:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762397.981435:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512947200:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762397.981439:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512947328:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762397.981452:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762397.981459:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762397/real 1789762397] req@ffff8dc0107aed80 x1876699512947328/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762397.981467:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512947328/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762397.981472:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762397.981474:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762397.981475:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512947328:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762397.981479:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512947456:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762397.981490:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762397.981496:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762397/real 1789762397] req@ffff8dc0107ad500 x1876699512947456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762397.981503:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512947456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762397.981508:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762397.981510:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762397.981513:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512947456:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762399.005048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762400.029197:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762401.053116:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762402.077044:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762403.101091:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762403.101104:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762403.101109:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762403.101121:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2377 00000100:00080000:2.0:1789762403.101132:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762403.101140:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789762403.101145:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762403.101179:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512947584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762403.101207:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762403.101212:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762403.101214:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762403.101219:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2377 00000100:00080000:2.0:1789762403.101224:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762403.101230:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762403.101239:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512947584:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789762403.101240:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762403.101276:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512947712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762403.101289:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762403.101301:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762403.101304:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762403.101309:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2377 00000100:00080000:2.0:1789762403.101314:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762403.101318:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762403.101338:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762403.101345:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762403.101360:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512947840/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762403.101372:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762403.101378:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762403/real 1789762403] req@ffff8dc00368d880 x1876699512947584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762403.101378:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762403.101383:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762403.101396:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512947584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762403.101398:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2377 00000100:00080000:2.0:1789762403.101406:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762403.101410:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762403.101414:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:0.0:1789762403.101417:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762403.101433:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512947584:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789762403.101436:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762403.101469:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512947968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762403.101472:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512947712:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762403.101497:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762403.101507:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762403/real 1789762403] req@ffff8dc00368fb80 x1876699512947712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762403.101516:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512947712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762403.101540:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762403.101542:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762403.101545:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512947712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762403.101549:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512947840:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762403.101562:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762403.101568:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762403/real 1789762403] req@ffff8dc00368f480 x1876699512947840/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762403.101576:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512947840/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762403.101582:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762403.101583:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762403.101585:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512947840:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762403.101595:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512947968:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762403.101606:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762403.101612:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762403/real 1789762403] req@ffff8dc00368f100 x1876699512947968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762403.101620:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512947968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762403.101625:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762403.101627:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762403.101629:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512947968:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762404.125040:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762405.149104:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762406.173043:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762407.197097:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762408.221025:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762408.221034:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762408.221038:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762408.221065:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2382 00000100:00080000:3.0:1789762408.221075:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762408.221081:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789762408.221083:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762408.221116:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512948096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762408.221137:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762408.221140:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762408.221142:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762408.221145:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2382 00000100:00080000:3.0:1789762408.221150:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762408.221153:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762408.221163:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762408.221167:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512948096:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762408.221175:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512948224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762408.221184:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762408.221187:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762408.221188:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762408.221191:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2382 00000100:00080000:3.0:1789762408.221194:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762408.221196:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762408.221200:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762408.221209:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512948352/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762408.221216:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762408.221219:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762408.221220:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789762408.221225:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762408.221228:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2382 00000100:00080000:3.0:1789762408.221232:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762408.221235:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762408.221240:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762408.221249:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512948480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762408.221276:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762408/real 1789762408] req@ffff8dc0107ac000 x1876699512948096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762408.221294:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512948096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762408.221302:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762408.221306:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762408.221309:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512948096:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762408.221333:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512948224:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762408.221368:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762408.221383:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762408/real 1789762408] req@ffff8dc0107ac700 x1876699512948224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762408.221397:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512948224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762408.221407:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762408.221411:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762408.221414:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512948224:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762408.221421:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512948352:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762408.221455:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762408.221467:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762408/real 1789762408] req@ffff8dc0107ae300 x1876699512948352/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762408.221481:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512948352/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762408.221489:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762408.221493:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762408.221497:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512948352:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762408.221503:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512948480:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762408.221522:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762408.221549:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762408/real 1789762408] req@ffff8dc0107aca80 x1876699512948480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762408.221571:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512948480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762408.221580:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762408.221584:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762408.221598:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512948480:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762409.245114:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762410.269137:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762411.293013:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762412.317055:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762413.341003:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762413.341011:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762413.341014:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762413.341023:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2387 00000100:00080000:1.0:1789762413.341031:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762413.341037:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789762413.341038:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789762413.341072:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512948608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762413.341099:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762413.341103:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762413.341105:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762413.341110:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2387 00000100:00080000:1.0:1789762413.341115:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762413.341121:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762413.341131:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762413.341132:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512948608:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762413.341149:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512948736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762413.341161:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762413.341165:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762413.341168:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762413.341172:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2387 00000100:00080000:1.0:1789762413.341186:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762413.341191:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762413.341198:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762413.341213:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512948864/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762413.341225:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762413.341231:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512948736:192.168.203.156@tcp:38:kworker.0 00000100:00080000:1.0:1789762413.341231:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762413.341236:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762413.341243:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2387 00000100:00080000:1.0:1789762413.341247:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762413.341252:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762413.341259:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762413.341263:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512948864:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762413.341277:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512948992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762413.341314:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512948992:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762414.365099:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762415.389058:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762416.413130:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762416.413217:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:1.0:1789762416.413262:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762416.413279:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762416.413325:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762416.413337:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762416.413347:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:0.0:1789762416.413375:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762413/real 1789762416] req@ffff8dc005c95500 x1876699512948736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762416.413404:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512948736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762416.413417:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762416.413423:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762416.413428:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512948736:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762416.413436:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762413/real 1789762416] req@ffff8dc005c95f80 x1876699512948864/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762416.413458:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512948864/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762416.413467:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762416.413471:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762416.413475:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512948864:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762416.413482:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762413/real 1789762416] req@ffff8dc005c94700 x1876699512948608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762416.413495:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512948608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762416.413504:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762416.413507:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762416.413511:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512948608:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762416.413536:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762413/real 1789762416] req@ffff8dc005c96a00 x1876699512948992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762416.413551:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512948992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762416.413557:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762416.413560:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762416.413562:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512948992:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762417.437036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762418.461059:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762418.461067:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762418.461071:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762418.461080:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2393 00000100:00080000:0.0:1789762418.461091:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762418.461097:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762418.461144:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b800 x1876699512949120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762418.461167:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762418.461170:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762418.461172:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762418.461176:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2393 00000100:00080000:0.0:1789762418.461184:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762418.461188:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762418.461196:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762418.461202:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512949120:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762418.461212:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514a300 x1876699512949248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762418.461222:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762418.461225:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762418.461227:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762418.461232:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2393 00000100:00080000:0.0:1789762418.461252:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762418.461256:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762418.461262:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762418.461275:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148e00 x1876699512949376/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762418.461284:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1789762418.461294:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762418.461298:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762418.461301:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762418.461307:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2393 00000100:00080000:0.0:1789762418.461311:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762418.461316:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762418.461322:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762418.461329:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762418/real 1789762418] req@ffff8dc00514b800 x1876699512949120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762418.461337:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b100 x1876699512949504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762418.461378:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b800 x1876699512949120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762418.461391:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762418.461396:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762418.461400:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512949120:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762418.461441:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512949248:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762418.461470:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00010000:00010000:3.0:1789762418.461478:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789762418.461489:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762418/real 1789762418] req@ffff8dc00514a300 x1876699512949248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762418.461504:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514a300 x1876699512949248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762418.461514:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762418.461518:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762418.461521:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512949248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762418.461528:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512949376:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762418.461552:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762418.461565:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762418/real 1789762418] req@ffff8dc005148e00 x1876699512949376/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762418.461578:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148e00 x1876699512949376/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762418.461595:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762418.461599:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762418.461603:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512949376:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762418.461610:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512949504:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762418.461633:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762418.461646:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762418/real 1789762418] req@ffff8dc00514b100 x1876699512949504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762418.461667:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b100 x1876699512949504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762418.461676:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762418.461680:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762418.461683:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512949504:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762419.485089:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762420.509038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762421.533027:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762422.557072:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762423.581030:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762423.581044:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762423.581050:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762423.581067:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2398 00000100:00080000:0.0:1789762423.581082:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762423.581096:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762423.581178:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149500 x1876699512949632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762423.581216:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762423.581221:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762423.581223:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762423.581228:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2398 00000100:00080000:0.0:1789762423.581233:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762423.581239:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762423.581252:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762423.581268:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b100 x1876699512949760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762423.581281:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512949632:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789762423.581283:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762423.581293:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762423.581296:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762423.581301:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2398 00000100:00080000:0.0:1789762423.581304:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762423.581308:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762423.581313:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762423.581344:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148e00 x1876699512949888/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762423.581354:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762423.581358:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762423.581360:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762423.581363:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2398 00000100:00080000:0.0:1789762423.581369:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762423.581373:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762423.581378:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762423.581388:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514a300 x1876699512950016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762423.581401:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00010000:00010000:0.0:1789762423.581487:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789762423.581493:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762423/real 1789762423] req@ffff8dc005149500 x1876699512949632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762423.581519:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149500 x1876699512949632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762423.581533:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762423.581537:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762423.581542:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512949632:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762423.581585:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512949760:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762423.581609:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762423.581623:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762423/real 1789762423] req@ffff8dc00514b100 x1876699512949760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762423.581635:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b100 x1876699512949760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762423.581647:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762423.581652:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762423.581657:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512949760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762423.581666:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512949888:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762423.581717:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762423.581736:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762423/real 1789762423] req@ffff8dc005148e00 x1876699512949888/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762423.581752:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148e00 x1876699512949888/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762423.581762:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762423.581765:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762423.581768:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512949888:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762423.581792:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512950016:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762423.581826:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762423.581864:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762423/real 1789762423] req@ffff8dc00514a300 x1876699512950016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762423.581903:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514a300 x1876699512950016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762423.581915:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762423.581978:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762423.581984:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512950016:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762424.605160:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762425.629395:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762426.653096:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762427.677065:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762428.701055:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762428.701066:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762428.701071:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762428.701084:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2403 00000100:00080000:2.0:1789762428.701096:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762428.701109:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789762428.701149:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762428.701160:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512950144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762428.701187:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762428.701190:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762428.701193:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762428.701197:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2403 00000100:00080000:2.0:1789762428.701200:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762428.701204:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762428.701213:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762428.701224:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512950272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762428.701235:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762428.701237:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512950144:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762428.701240:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762428.701246:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762428.701262:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2403 00000100:00080000:2.0:1789762428.701265:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762428.701267:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762428.701271:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762428.701282:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512950400/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762428.701295:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762428.701298:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762428.701300:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762428.701303:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2403 00000100:00080000:2.0:1789762428.701306:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762428.701309:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762428.701312:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762428.701320:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512950528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762428.701334:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762428.701402:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762428/real 1789762428] req@ffff8dc00368ce00 x1876699512950144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762428.701433:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512950144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762428.701448:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762428.701452:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762428.701457:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512950144:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762428.701485:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512950272:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762428.701513:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762428.701527:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762428/real 1789762428] req@ffff8dc00368ea00 x1876699512950272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762428.701539:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512950272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762428.701554:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762428.701557:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762428.701560:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512950272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762428.701566:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512950400:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762428.701583:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762428.701593:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762428/real 1789762428] req@ffff8dc00368c000 x1876699512950400/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762428.701603:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512950400/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762428.701612:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762428.701614:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762428.701617:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512950400:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762428.701623:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512950528:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762428.701638:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762428.701647:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762428/real 1789762428] req@ffff8dc00368ca80 x1876699512950528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762428.701657:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512950528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762428.701665:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762428.701668:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762428.701671:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512950528:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762429.725056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762430.749110:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762431.773112:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762432.797032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762433.820978:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762433.820990:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762433.820996:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762433.821009:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2408 00000100:00080000:2.0:1789762433.821022:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762433.821031:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762433.821077:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512950656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1789762433.821122:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762433.821134:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762433.821141:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762433.821144:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762433.821150:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2408 00000100:00080000:2.0:1789762433.821157:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762433.821164:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762433.821177:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762433.821215:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512950656:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762433.821218:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512950784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762433.821239:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762433.821242:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762433.821244:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762433.821249:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2408 00000100:00080000:2.0:1789762433.821253:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762433.821257:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762433.821262:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762433.821275:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512950912/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762433.821300:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762433.821304:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762433.821307:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762433.821311:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2408 00000100:00080000:2.0:1789762433.821316:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789762433.821321:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762433.821328:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762433.821335:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762433.821350:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512951040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762433.821385:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762433/real 1789762433] req@ffff8dc00368f800 x1876699512950656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762433.821409:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512950656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762433.821421:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762433.821426:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762433.821431:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512950656:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762433.821459:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512950784:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762433.821482:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762433.821496:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762433/real 1789762433] req@ffff8dc00368ed80 x1876699512950784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762433.821509:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512950784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762433.821518:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762433.821521:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762433.821525:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512950784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762433.821531:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512950912:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762433.821548:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762433.821559:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762433/real 1789762433] req@ffff8dc00368e300 x1876699512950912/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762433.821572:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512950912/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762433.821580:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762433.821583:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762433.821586:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512950912:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762433.821592:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512951040:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762433.821609:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762433.821619:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762433/real 1789762433] req@ffff8dc00368df80 x1876699512951040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762433.821631:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512951040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762433.821639:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762433.821642:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762433.821645:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512951040:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762434.845091:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762435.869101:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762436.893192:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762437.917099:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762438.941008:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762438.941017:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762438.941021:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762438.941031:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2413 00000100:00080000:3.0:1789762438.941040:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762438.941046:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762438.941095:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699512951168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1789762438.941116:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762438.941120:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762438.941126:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762438.941130:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762438.941134:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2413 00000100:00080000:3.0:1789762438.941138:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762438.941144:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762438.941154:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762438.941167:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699512951296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762438.941168:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512951168:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762438.941181:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762438.941184:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762438.941185:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762438.941188:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2413 00000100:00080000:3.0:1789762438.941191:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762438.941193:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762438.941197:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762438.941207:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512951424/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762438.941214:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762438.941217:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762438.941218:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762438.941221:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2413 00000100:00080000:3.0:1789762438.941224:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789762438.941229:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762438.941231:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762438.941239:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762438.941252:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512951552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762438.941276:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762438/real 1789762438] req@ffff8dc0107adc00 x1876699512951168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762438.941303:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699512951168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762438.941313:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762438.941316:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762438.941319:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512951168:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762438.941358:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512951296:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762438.941376:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762438.941386:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762438/real 1789762438] req@ffff8dc0107afb80 x1876699512951296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762438.941395:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699512951296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762438.941418:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762438.941421:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762438.941425:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512951296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762438.941430:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512951424:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762438.941447:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762438.941456:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762438/real 1789762438] req@ffff8dc0107aca80 x1876699512951424/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762438.941467:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512951424/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762438.941474:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762438.941477:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762438.941480:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512951424:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762438.941486:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512951552:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762438.941500:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762438.941508:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762438/real 1789762438] req@ffff8dc0107ae300 x1876699512951552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762438.941518:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512951552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762438.941525:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762438.941528:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762438.941548:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512951552:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762439.965116:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762440.989134:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762442.013033:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762443.037114:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762444.061000:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762444.061011:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762444.061016:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762444.061058:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2418 00010000:00010000:0.0:1789762444.061070:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762444.061071:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762444.061083:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762444.061140:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512951680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762444.061170:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762444.061187:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762444.061190:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762444.061196:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2418 00000100:00080000:3.0:1789762444.061202:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762444.061208:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762444.061221:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762444.061240:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512951680:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762444.061241:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512951808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762444.061257:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762444.061261:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762444.061280:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762444.061283:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2418 00000100:00080000:3.0:1789762444.061287:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762444.061290:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762444.061295:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762444.061306:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512951936/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762444.061315:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762444.061318:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762444.061321:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762444.061325:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2418 00000800:00000100:1.0:1789762444.061326:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762444.061328:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762444.061332:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762444.061337:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762444.061347:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512952064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762444.061386:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762444/real 1789762444] req@ffff8dc0107ac700 x1876699512951680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762444.061409:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512951680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762444.061419:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762444.061422:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762444.061425:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512951680:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762444.061456:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512951808:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762444.061477:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762444.061487:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762444/real 1789762444] req@ffff8dc0107ac000 x1876699512951808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762444.061497:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512951808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762444.061503:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762444.061505:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762444.061508:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512951808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762444.061512:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512951936:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762444.061524:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762444.061530:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762444/real 1789762444] req@ffff8dc0107ad500 x1876699512951936/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762444.061538:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512951936/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762444.061544:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762444.061546:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762444.061548:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512951936:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762444.061559:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512952064:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762444.061571:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762444.061577:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762444/real 1789762444] req@ffff8dc0107aed80 x1876699512952064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762444.061586:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512952064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762444.061597:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762444.061599:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762444.061602:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512952064:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762445.085030:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762446.109177:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762447.133165:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762448.157177:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762449.181150:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762449.181163:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762449.181169:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762449.181183:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2423 00010000:00010000:1.0:1789762449.181193:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762449.181210:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762449.181221:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762449.181273:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512952192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762449.181323:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762449.181328:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762449.181330:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762449.181336:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2423 00000100:00080000:2.0:1789762449.181340:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762449.181346:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762449.181359:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762449.181376:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512952320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762449.181393:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512952192:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762449.181395:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762449.181409:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762449.181416:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762449.181423:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2423 00000100:00080000:2.0:1789762449.181427:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762449.181431:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762449.181439:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762449.181456:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512952448/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762449.181469:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762449.181473:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762449.181476:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762449.181482:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2423 00000100:00080000:2.0:1789762449.181487:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762449.181493:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762449.181499:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762449.181513:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512952576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762449.181532:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762449.181609:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762449/real 1789762449] req@ffff8dc00368c700 x1876699512952192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762449.181636:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512952192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762449.181677:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762449.181686:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762449.181695:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512952192:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762449.181749:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512952320:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762449.181796:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762449.181817:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762449/real 1789762449] req@ffff8dc00368dc00 x1876699512952320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762449.181834:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512952320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762449.181846:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762449.181850:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762449.181856:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512952320:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762449.181865:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512952448:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762449.181970:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762449.181998:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762449/real 1789762449] req@ffff8dc00368e680 x1876699512952448/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762449.182023:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512952448/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762449.182039:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762449.182047:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762449.182055:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512952448:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762449.182068:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512952576:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762449.182109:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762449.182126:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762449/real 1789762449] req@ffff8dc00368d180 x1876699512952576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762449.182143:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512952576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762449.182153:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762449.182158:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762449.182162:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512952576:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762450.205100:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762451.229143:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762452.253156:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762453.277125:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762454.301022:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762454.301030:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762454.301034:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762454.301044:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2428 00010000:00010000:0.0:1789762454.301052:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762454.301052:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762454.301061:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762454.301100:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512952704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762454.301123:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762454.301127:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762454.301130:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762454.301135:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2428 00000100:00080000:3.0:1789762454.301140:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762454.301146:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762454.301156:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762454.301165:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512952704:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762454.301183:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512952832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762454.301195:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762454.301198:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762454.301199:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762454.301203:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2428 00000100:00080000:3.0:1789762454.301207:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762454.301211:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762454.301216:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762454.301226:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512952960/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762454.301236:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762454.301249:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762454.301251:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789762454.301256:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762454.301259:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2428 00000100:00080000:3.0:1789762454.301265:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762454.301268:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762454.301272:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762454.301280:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512953088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762454.301299:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762454/real 1789762454] req@ffff8dc0107ad180 x1876699512952704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762454.301328:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512952704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762454.301339:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762454.301344:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762454.301348:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512952704:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762454.301374:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512952832:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762454.301396:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762454.301408:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762454/real 1789762454] req@ffff8dc0107af480 x1876699512952832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762454.301427:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512952832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762454.301437:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762454.301440:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762454.301443:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512952832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762454.301451:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512952960:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762454.301472:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762454.301487:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762454/real 1789762454] req@ffff8dc0107adf80 x1876699512952960/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762454.301502:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512952960/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762454.301512:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762454.301516:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762454.301521:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512952960:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762454.301528:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512953088:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762454.301551:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762454.301568:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762454/real 1789762454] req@ffff8dc0107ac380 x1876699512953088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762454.301582:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512953088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762454.301596:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762454.301600:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762454.301604:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512953088:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762455.325143:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762456.349081:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762457.373023:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762458.397075:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762459.421021:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762459.421032:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762459.421037:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1789762459.421050:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762459.421056:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2434 00000100:00080000:3.0:1789762459.421067:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762459.421074:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762459.421105:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512953216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762459.421125:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762459.421128:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762459.421130:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762459.421133:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2434 00000100:00080000:3.0:1789762459.421136:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762459.421138:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762459.421145:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762459.421154:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512953344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762459.421161:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762459.421164:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762459.421165:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762459.421168:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2434 00000100:00080000:3.0:1789762459.421170:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762459.421172:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762459.421175:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762459.421180:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512953216:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762459.421186:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512953472/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762459.421212:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762459.421214:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762459.421216:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762459.421218:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2434 00000100:00080000:3.0:1789762459.421221:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762459.421224:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762459.421226:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762459.421236:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512953600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762459.421281:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762459.421331:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762459/real 1789762459] req@ffff8dc0107af800 x1876699512953216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762459.421360:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512953216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762459.421372:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762459.421377:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762459.421381:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512953216:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762459.421389:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512953344:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762459.421412:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762459.421425:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762459/real 1789762459] req@ffff8dc0107ae680 x1876699512953344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762459.421438:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512953344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762459.421459:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762459.421463:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762459.421468:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512953344:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762459.421491:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512953472:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762459.421520:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762459.421531:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762459/real 1789762459] req@ffff8dc0107ace00 x1876699512953472/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762459.421543:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512953472/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762459.421553:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762459.421556:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762459.421560:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512953472:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762459.421566:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512953600:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762459.421601:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762459.421613:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762459/real 1789762459] req@ffff8dc0107af100 x1876699512953600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762459.421636:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512953600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762459.421645:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762459.421648:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762459.421652:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512953600:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762460.445061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762461.469118:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762462.493046:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762463.517017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762464.541039:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1789762464.541045:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762464.541051:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762464.541058:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762464.541073:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2439 00000100:00080000:3.0:1789762464.541085:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762464.541094:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762464.541139:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512953728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762464.541175:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762464.541203:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762464.541206:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762464.541212:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2439 00000100:00080000:3.0:1789762464.541218:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762464.541224:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762464.541241:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512953728:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789762464.541251:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762464.541273:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512953856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762464.541286:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762464.541291:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762464.541294:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762464.541310:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2439 00000100:00080000:3.0:1789762464.541319:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762464.541326:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789762464.541335:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789762464.541340:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762464.541385:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512953984/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762464.541405:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762464.541410:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762464.541413:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789762464.541419:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762464/real 1789762464] req@ffff8dc0107aea00 x1876699512953728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762464.541422:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2439 00000100:00080000:3.0:1789762464.541432:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762464.541443:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762464.541455:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762464.541464:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512953728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762464.541480:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789762464.541481:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512954112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762464.541490:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762464.541503:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512953728:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762464.541587:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512953856:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762464.541634:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762464.541650:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762464/real 1789762464] req@ffff8dc0107ad880 x1876699512953856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762464.541672:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512953856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762464.541680:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762464.541683:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762464.541686:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512953856:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762464.541692:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512953984:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762464.541710:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762464.541718:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762464/real 1789762464] req@ffff8dc0107af100 x1876699512953984/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762464.541729:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512953984/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762464.541736:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762464.541739:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762464.541741:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512953984:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762464.541745:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512954112:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762464.541758:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762464.541765:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762464/real 1789762464] req@ffff8dc0107ace00 x1876699512954112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762464.541775:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512954112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762464.541781:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762464.541783:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762464.541786:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512954112:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762465.565047:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762466.589033:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762467.613049:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762468.637110:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762469.660969:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762469.660981:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762469.660985:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762469.660997:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2444 00000100:00080000:3.0:1789762469.661008:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762469.661015:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789762469.661041:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762469.661050:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512954240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762469.661066:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762469.661069:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762469.661071:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762469.661075:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2444 00000100:00080000:3.0:1789762469.661078:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762469.661081:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762469.661088:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762469.661098:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512954240:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762469.661099:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512954368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762469.661113:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762469.661116:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762469.661117:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762469.661119:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2444 00000100:00080000:3.0:1789762469.661121:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762469.661123:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762469.661127:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762469.661135:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512954496/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762469.661144:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762469.661147:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762469.661148:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762469.661151:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2444 00000100:00080000:3.0:1789762469.661153:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762469.661155:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762469.661158:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762469.661165:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512954624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762469.661175:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762469.661218:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762469/real 1789762469] req@ffff8dc0107ae680 x1876699512954240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762469.661246:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512954240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762469.661257:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762469.661261:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762469.661266:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512954240:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762469.661291:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512954368:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762469.661314:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762469.661329:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762469/real 1789762469] req@ffff8dc0107af800 x1876699512954368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762469.661342:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512954368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762469.661351:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762469.661355:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762469.661358:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512954368:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762469.661365:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512954496:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762469.661386:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762469.661398:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762469/real 1789762469] req@ffff8dc0107ac380 x1876699512954496/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762469.661410:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512954496/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762469.661426:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762469.661429:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762469.661433:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512954496:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762469.661439:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512954624:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762469.661461:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762469.661472:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762469/real 1789762469] req@ffff8dc0107adf80 x1876699512954624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762469.661485:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512954624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762469.661509:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762469.661514:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762469.661518:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512954624:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762470.685061:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762471.709102:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762472.733041:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762473.757080:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762474.781082:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762474.781096:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762474.781101:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762474.781116:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2449 00000100:00080000:3.0:1789762474.781130:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762474.781141:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789762474.781175:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762474.781242:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512954752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762474.781284:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762474.781289:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762474.781292:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762474.781298:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2449 00000100:00080000:3.0:1789762474.781302:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762474.781308:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762474.781335:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762474.781353:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512954880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762474.781368:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762474.781372:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512954752:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762474.781374:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762474.781381:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762474.781393:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2449 00000100:00080000:3.0:1789762474.781399:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762474.781406:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762474.781413:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762474.781437:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512955008/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762474.781452:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789762474.781483:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762474.781491:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762474.781495:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762474.781503:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2449 00000100:00080000:3.0:1789762474.781516:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762474.781523:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762474.781531:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762474.781532:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762474/real 1789762474] req@ffff8dc0107af480 x1876699512954752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762474.781561:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512955136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762474.781570:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512954752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762474.781602:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762474.781609:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762474.781613:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512954752:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762474.781640:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512954880:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762474.781665:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762474.781677:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762474/real 1789762474] req@ffff8dc0107ad180 x1876699512954880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762474.781688:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512954880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762474.781695:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762474.781698:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762474.781701:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512954880:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762474.781706:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512955008:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762474.781721:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762474.781729:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762474/real 1789762474] req@ffff8dc0107aed80 x1876699512955008/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762474.781738:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512955008/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762474.781745:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762474.781747:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762474.781750:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512955008:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762474.781756:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512955136:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762474.781789:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762474.781805:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762474/real 1789762474] req@ffff8dc0107ad500 x1876699512955136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762499 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762474.781823:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512955136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762499 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762474.781831:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762474.781834:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762474.781837:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512955136:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762475.805071:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762476.829087:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762477.853092:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762478.877045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762479.901007:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762479.901019:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762479.901023:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789762479.901024:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762479.901038:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2454 00000100:00080000:2.0:1789762479.901047:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762479.901054:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762479.901090:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512955264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762479.901111:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762479.901114:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762479.901116:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762479.901119:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2454 00000100:00080000:2.0:1789762479.901122:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762479.901125:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762479.901133:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762479.901142:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512955264:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762479.901144:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512955392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762479.901157:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762479.901160:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762479.901161:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762479.901164:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2454 00000100:00080000:2.0:1789762479.901166:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762479.901169:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762479.901172:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762479.901181:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512955520/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762479.901188:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762479.901191:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762479.901192:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762479.901195:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2454 00000100:00080000:2.0:1789762479.901200:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762479.901203:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762479.901206:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762479.901213:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512955648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762479.901252:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512955392:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762479.901280:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512955520:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762479.901295:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512955648:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762480.925052:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762481.949037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762482.973047:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762482.973153:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:1.0:1789762482.973191:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762482.973205:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762482.973237:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762482.973247:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762482.973262:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:0.0:1789762482.973298:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762479/real 1789762482] req@ffff8dc00368e680 x1876699512955392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762482.973328:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512955392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762482.973341:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762482.973347:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762482.973352:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512955392:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762482.973361:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762479/real 1789762482] req@ffff8dc00368dc00 x1876699512955520/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762482.973373:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512955520/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762482.973381:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762482.973385:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762482.973389:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512955520:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762482.973411:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762479/real 1789762482] req@ffff8dc00368c700 x1876699512955648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762482.973425:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512955648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762482.973434:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762482.973437:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762482.973441:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512955648:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762482.973460:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762479/real 1789762482] req@ffff8dc00368d180 x1876699512955264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762482.973472:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512955264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762482.973485:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762482.973488:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762482.973491:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512955264:192.168.203.156@tcp:250:kworker.0 00010000:00010000:1.0:1789762483.997070:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762485.021014:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762485.021027:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762485.021034:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762485.021046:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2459 00000100:00080000:3.0:1789762485.021059:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762485.021068:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789762485.021083:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762485.021114:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512955776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762485.021146:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762485.021150:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762485.021152:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762485.021158:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2459 00000100:00080000:3.0:1789762485.021163:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762485.021170:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762485.021181:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762485.021196:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512955776:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762485.021201:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512955904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762485.021217:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762485.021221:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762485.021223:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762485.021228:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2459 00000100:00080000:3.0:1789762485.021231:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762485.021235:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762485.021241:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762485.021256:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512956032/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762485.021274:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789762485.021285:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762485.021285:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762485.021290:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762485.021297:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2459 00000100:00080000:3.0:1789762485.021302:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762485.021307:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762485.021313:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762485.021329:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512956160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762485.021345:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762485/real 1789762485] req@ffff8dc0107ac000 x1876699512955776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762485.021370:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512955776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762485.021397:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762485.021403:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762485.021408:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512955776:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762485.021437:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512955904:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762485.021484:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762485.021498:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762485/real 1789762485] req@ffff8dc0107ac700 x1876699512955904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762485.021513:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512955904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762485.021523:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762485.021527:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762485.021531:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512955904:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762485.021539:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512956032:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762485.021565:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762485.021577:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762485/real 1789762485] req@ffff8dc0107ae300 x1876699512956032/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762485.021591:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512956032/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762485.021601:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762485.021605:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762485.021609:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512956032:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762485.021617:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512956160:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762485.021641:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762485.021654:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762485/real 1789762485] req@ffff8dc0107aca80 x1876699512956160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762485.021666:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512956160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762485.021675:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762485.021678:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762485.021682:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512956160:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762486.045126:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762487.069042:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762488.093015:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762489.116990:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762490.141001:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762490.141009:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762490.141012:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762490.141021:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2464 00000100:00080000:3.0:1789762490.141031:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762490.141036:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789762490.141066:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762490.141071:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699512956288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762490.141093:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762490.141096:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762490.141098:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762490.141101:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2464 00000100:00080000:3.0:1789762490.141103:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762490.141105:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762490.141113:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762490.141122:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699512956416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762490.141129:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762490.141132:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762490.141133:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762490.141136:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2464 00000100:00080000:3.0:1789762490.141138:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762490.141140:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762490.141142:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762490.141152:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512956544/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762490.141157:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512956288:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762490.141164:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762490.141167:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762490.141168:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762490.141170:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2464 00000100:00080000:3.0:1789762490.141172:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762490.141175:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762490.141178:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762490.141185:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512956672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762490.141219:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762490.141257:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762490/real 1789762490] req@ffff8dc0107afb80 x1876699512956288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762490.141272:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699512956288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762490.141280:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762490.141283:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762490.141287:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512956288:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762490.141292:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512956416:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762490.141307:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762490.141316:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762490/real 1789762490] req@ffff8dc0107adc00 x1876699512956416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762490.141327:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699512956416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762490.141335:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762490.141338:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762490.141354:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512956416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762490.141381:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512956544:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762490.141405:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762490.141416:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762490/real 1789762490] req@ffff8dc0107aca80 x1876699512956544/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762490.141427:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512956544/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762490.141432:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762490.141435:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762490.141437:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512956544:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762490.141445:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512956672:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762490.141467:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762490.141480:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762490/real 1789762490] req@ffff8dc0107ae300 x1876699512956672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762490.141489:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512956672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762490.141495:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762490.141498:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762490.141500:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512956672:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762491.165064:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762492.189028:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762493.213054:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762494.237020:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762495.261013:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762495.261038:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762495.261043:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762495.261046:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762495.261055:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2469 00000100:00080000:0.0:1789762495.261064:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762495.261069:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762495.261100:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b800 x1876699512956800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762495.261122:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762495.261125:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762495.261127:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762495.261130:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2469 00000100:00080000:0.0:1789762495.261133:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762495.261136:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762495.261143:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762495.261152:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149f80 x1876699512956928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762495.261161:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762495.261162:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512956800:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789762495.261164:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762495.261168:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762495.261173:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2469 00000100:00080000:0.0:1789762495.261175:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762495.261177:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762495.261180:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762495.261189:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149180 x1876699512957056/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762495.261197:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762495.261201:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762495.261202:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762495.261205:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2469 00000100:00080000:0.0:1789762495.261220:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789762495.261224:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1789762495.261225:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762495.261232:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762495.261241:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149880 x1876699512957184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762495.261264:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762495/real 1789762495] req@ffff8dc00514b800 x1876699512956800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762495.261280:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b800 x1876699512956800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762495.261288:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762495.261291:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762495.261294:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512956800:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762495.261330:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512956928:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762495.261348:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762495.261358:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762495/real 1789762495] req@ffff8dc005149f80 x1876699512956928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762495.261367:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149f80 x1876699512956928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762495.261373:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762495.261375:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762495.261377:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512956928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762495.261381:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512957056:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762495.261392:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762495.261398:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762495/real 1789762495] req@ffff8dc005149180 x1876699512957056/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762495.261405:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149180 x1876699512957056/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762495.261410:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762495.261411:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762495.261413:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512957056:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762495.261417:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512957184:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762495.261428:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762495.261434:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762495/real 1789762495] req@ffff8dc005149880 x1876699512957184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762495.261441:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149880 x1876699512957184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762495.261446:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762495.261448:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762495.261450:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512957184:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762496.285071:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762497.309084:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762498.333033:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762499.357066:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762500.381040:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762500.381052:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762500.381057:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762500.381069:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2474 00000100:00080000:0.0:1789762500.381081:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762500.381089:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762500.381148:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149c00 x1876699512957312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762500.381187:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762500.381190:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762500.381192:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762500.381195:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2474 00000100:00080000:0.0:1789762500.381198:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762500.381201:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762500.381210:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762500.381222:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148380 x1876699512957440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762500.381227:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512957312:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789762500.381232:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762500.381235:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762500.381237:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762500.381240:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2474 00000100:00080000:0.0:1789762500.381242:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762500.381245:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762500.381248:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762500.381259:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148000 x1876699512957568/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762500.381287:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762500.381291:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762500.381293:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762500.381298:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2474 00000100:00080000:0.0:1789762500.381330:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762500.381336:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762500.381342:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762500.381354:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762500.381356:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b480 x1876699512957696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762500.381409:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762500/real 1789762500] req@ffff8dc005149c00 x1876699512957312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762500.381428:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149c00 x1876699512957312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762500.381439:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762500.381442:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762500.381455:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512957312:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762500.381482:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512957440:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762500.381507:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00010000:00010000:2.0:1789762500.381509:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789762500.381524:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762500/real 1789762500] req@ffff8dc005148380 x1876699512957440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762500.381541:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148380 x1876699512957440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762500.381550:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762500.381555:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762500.381559:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512957440:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762500.381566:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512957568:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762500.381588:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762500.381606:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762500/real 1789762500] req@ffff8dc005148000 x1876699512957568/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762500.381618:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148000 x1876699512957568/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762500.381626:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762500.381630:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762500.381633:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512957568:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762500.381639:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512957696:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762500.381658:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762500.381669:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762500/real 1789762500] req@ffff8dc00514b480 x1876699512957696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762500.381681:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b480 x1876699512957696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762500.381689:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762500.381692:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762500.381696:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512957696:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762501.405025:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762502.429097:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762503.453032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762504.477036:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762505.501016:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762505.501025:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762505.501029:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762505.501038:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2480 00000100:00080000:3.0:1789762505.501047:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762505.501054:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789762505.501068:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762505.501087:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512957824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762505.501111:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762505.501114:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762505.501115:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762505.501118:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2480 00000100:00080000:3.0:1789762505.501121:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762505.501123:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762505.501131:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762505.501141:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512957952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762505.501149:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512957824:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762505.501152:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762505.501157:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762505.501158:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762505.501162:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2480 00000100:00080000:3.0:1789762505.501164:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762505.501166:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762505.501172:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762505.501179:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512958080/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762505.501187:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762505.501189:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762505.501191:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762505.501194:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2480 00000100:00080000:3.0:1789762505.501197:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762505.501201:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762505.501204:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762505.501211:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762505.501214:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512958208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762505.501267:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762505/real 1789762505] req@ffff8dc0107ac700 x1876699512957824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762505.501282:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512957824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762505.501290:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762505.501293:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762505.501296:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512957824:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762505.501314:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512957952:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762505.501330:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762505.501338:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762505/real 1789762505] req@ffff8dc0107ac000 x1876699512957952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762505.501346:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512957952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762505.501352:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762505.501354:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762505.501356:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512957952:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762505.501360:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512958080:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762505.501371:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762505.501377:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762505/real 1789762505] req@ffff8dc0107ad500 x1876699512958080/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762505.501390:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512958080/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762505.501395:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762505.501397:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762505.501399:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512958080:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762505.501402:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512958208:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762505.501413:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762505.501419:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762505/real 1789762505] req@ffff8dc0107aed80 x1876699512958208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762505.501427:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512958208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762505.501432:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762505.501433:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762505.501435:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512958208:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762506.525049:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762507.549077:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762508.573057:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762509.597089:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762510.620998:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762510.621009:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762510.621015:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762510.621027:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2485 00010000:00010000:0.0:1789762510.621029:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762510.621040:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762510.621048:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762510.621092:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512958336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762510.621116:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762510.621120:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762510.621122:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762510.621127:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2485 00000100:00080000:3.0:1789762510.621132:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762510.621137:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762510.621148:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512958336:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789762510.621149:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762510.621168:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512958464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762510.621177:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762510.621180:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762510.621181:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762510.621184:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2485 00000100:00080000:3.0:1789762510.621200:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762510.621204:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762510.621208:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762510.621218:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512958592/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762510.621226:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762510.621229:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762510.621230:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762510.621233:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2485 00000100:00080000:3.0:1789762510.621236:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762510.621239:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1789762510.621243:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789762510.621244:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762510.621254:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512958720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762510.621291:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762510/real 1789762510] req@ffff8dc0107ad180 x1876699512958336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762510.621307:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512958336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762510.621316:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762510.621319:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762510.621323:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512958336:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762510.621345:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512958464:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762510.621359:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762510.621368:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762510/real 1789762510] req@ffff8dc0107af480 x1876699512958464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762510.621376:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512958464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762510.621382:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762510.621384:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762510.621386:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512958464:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762510.621389:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512958592:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762510.621400:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762510.621406:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762510/real 1789762510] req@ffff8dc0107adf80 x1876699512958592/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762510.621413:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512958592/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762510.621418:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762510.621420:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762510.621422:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512958592:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762510.621426:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512958720:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762510.621438:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762510.621443:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762510/real 1789762510] req@ffff8dc0107ac380 x1876699512958720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762510.621451:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512958720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762510.621455:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762510.621457:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762510.621459:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512958720:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762511.645053:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762512.669086:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762513.693013:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762514.717054:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762515.741003:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762515.741014:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762515.741019:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762515.741031:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2490 00000100:00080000:0.0:1789762515.741041:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762515.741048:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762515.741087:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514bb80 x1876699512958848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762515.741117:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762515.741121:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762515.741123:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762515.741128:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2490 00000100:00080000:0.0:1789762515.741137:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762515.741142:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762515.741162:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762515.741179:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514ad80 x1876699512958976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762515.741191:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762515.741195:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762515.741200:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762515.741201:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512958848:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789762515.741207:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2490 00000100:00080000:0.0:1789762515.741213:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762515.741217:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762515.741224:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762515.741238:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148700 x1876699512959104/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762515.741250:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762515.741254:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762515.741256:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762515.741261:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2490 00000100:00080000:0.0:1789762515.741265:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762515.741270:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1789762515.741275:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:0.0:1789762515.741276:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762515.741291:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514aa00 x1876699512959232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762515.741322:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762515/real 1789762515] req@ffff8dc00514bb80 x1876699512958848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762515.741345:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514bb80 x1876699512958848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762515.741359:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762515.741365:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00010000:00010000:2.0:1789762515.741365:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789762515.741373:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512958848:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762515.741399:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512958976:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762515.741423:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762515.741436:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762515/real 1789762515] req@ffff8dc00514ad80 x1876699512958976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762515.741450:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514ad80 x1876699512958976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762515.741459:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762515.741463:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762515.741466:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512958976:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762515.741473:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512959104:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762515.741493:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762515.741502:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762515/real 1789762515] req@ffff8dc005148700 x1876699512959104/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762515.741511:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148700 x1876699512959104/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762515.741517:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762515.741519:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762515.741521:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512959104:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762515.741524:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512959232:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762515.741536:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762515.741542:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762515/real 1789762515] req@ffff8dc00514aa00 x1876699512959232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762540 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762515.741554:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514aa00 x1876699512959232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762540 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762515.741560:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762515.741561:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762515.741563:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512959232:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762516.765025:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762517.789058:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762518.813259:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762519.837060:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762520.861007:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762520.861018:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762520.861023:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762520.861034:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2495 00000100:00080000:3.0:1789762520.861044:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762520.861050:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789762520.861052:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762520.861089:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512959360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762520.861122:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762520.861126:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762520.861129:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762520.861133:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2495 00000100:00080000:3.0:1789762520.861138:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762520.861143:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762520.861152:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762520.861161:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512959360:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762520.861166:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512959488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762520.861176:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762520.861179:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762520.861181:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762520.861184:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2495 00000100:00080000:3.0:1789762520.861188:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762520.861191:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762520.861197:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762520.861207:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512959616/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762520.861216:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1789762520.861219:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762520.861219:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762520.861221:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762520.861225:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2495 00000100:00080000:3.0:1789762520.861227:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762520.861231:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762520.861235:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762520.861244:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512959744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762520.861255:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762520/real 1789762520] req@ffff8dc0107af800 x1876699512959360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762520.861273:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512959360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762520.861285:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762520.861289:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762520.861293:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512959360:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762520.861312:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512959488:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762520.861331:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762520.861341:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762520/real 1789762520] req@ffff8dc0107ae680 x1876699512959488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762520.861349:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512959488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762520.861355:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762520.861357:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762520.861359:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512959488:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762520.861363:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512959616:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762520.861373:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762520.861379:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762520/real 1789762520] req@ffff8dc0107ace00 x1876699512959616/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762520.861386:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512959616/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762520.861392:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762520.861394:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762520.861396:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512959616:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762520.861399:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512959744:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762520.861410:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762520.861419:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762520/real 1789762520] req@ffff8dc0107af100 x1876699512959744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762520.861431:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512959744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762520.861439:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762520.861442:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762520.861446:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512959744:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762521.885075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762522.908993:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762523.933030:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762524.957031:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762525.980997:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762525.981019:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762525.981025:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762525.981029:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762525.981038:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2500 00000100:00080000:3.0:1789762525.981046:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762525.981052:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762525.981100:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512959872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762525.981119:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762525.981122:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762525.981124:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762525.981127:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2500 00000100:00080000:3.0:1789762525.981132:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1789762525.981137:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512959872:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762525.981138:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762525.981150:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762525.981164:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512960000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762525.981172:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762525.981175:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762525.981177:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762525.981181:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2500 00000100:00080000:3.0:1789762525.981184:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762525.981187:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762525.981191:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789762525.981200:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762525.981203:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512960128/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762525.981212:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762525.981215:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762525.981217:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762525.981220:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2500 00000100:00080000:3.0:1789762525.981222:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762525.981225:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762525.981229:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762525.981240:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762525/real 1789762525] req@ffff8dc0107ad880 x1876699512959872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762525.981241:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512960256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762525.981261:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512959872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762525.981269:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762525.981272:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762525.981275:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512959872:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762525.981294:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512960000:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762525.981311:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762525.981324:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762525/real 1789762525] req@ffff8dc0107aea00 x1876699512960000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762525.981334:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512960000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762525.981340:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762525.981342:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762525.981345:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512960000:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762525.981351:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512960128:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762525.981362:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762525.981368:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762525/real 1789762525] req@ffff8dc0107af100 x1876699512960128/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762525.981375:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512960128/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762525.981380:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762525.981382:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762525.981384:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512960128:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762525.981392:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512960256:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762525.981403:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762525.981411:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762525/real 1789762525] req@ffff8dc0107ace00 x1876699512960256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762525.981418:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512960256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762525.981423:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762525.981425:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762525.981427:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512960256:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762527.004992:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762528.029005:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762529.053049:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762530.077039:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762531.101004:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762531.101023:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762531.101029:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762531.101033:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762531.101053:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2505 00000100:00080000:3.0:1789762531.101061:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762531.101066:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762531.101105:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512960384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762531.101125:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762531.101128:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762531.101130:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762531.101133:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2505 00000100:00080000:3.0:1789762531.101135:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762531.101139:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762531.101147:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762531.101160:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512960384:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762531.101160:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512960512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762531.101174:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762531.101176:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762531.101177:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762531.101180:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2505 00000100:00080000:3.0:1789762531.101182:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762531.101186:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762531.101189:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762531.101199:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512960640/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762531.101205:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762531.101208:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762531.101209:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762531.101212:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2505 00000100:00080000:3.0:1789762531.101215:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762531.101217:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762531.101220:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762531.101224:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762531.101231:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512960768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762531.101266:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762531/real 1789762531] req@ffff8dc0107ae680 x1876699512960384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762531.101286:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512960384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762531.101297:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762531.101300:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762531.101303:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512960384:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762531.101322:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512960512:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762531.101340:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762531.101349:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762531/real 1789762531] req@ffff8dc0107af800 x1876699512960512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762531.101357:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512960512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762531.101363:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762531.101365:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762531.101367:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512960512:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762531.101370:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512960640:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762531.101382:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762531.101392:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762531/real 1789762531] req@ffff8dc0107ac380 x1876699512960640/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762531.101411:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512960640/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762531.101421:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762531.101424:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762531.101428:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512960640:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762531.101434:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512960768:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762531.101452:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762531.101460:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762531/real 1789762531] req@ffff8dc0107adf80 x1876699512960768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762531.101468:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512960768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762531.101474:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762531.101476:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762531.101478:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512960768:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762532.125100:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762533.149079:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762534.173034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762535.197073:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762536.221085:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762536.221140:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762536.221148:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762536.221152:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762536.221163:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2510 00000100:00080000:3.0:1789762536.221174:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762536.221181:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762536.221220:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512960896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762536.221248:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762536.221252:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762536.221255:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762536.221261:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2510 00000100:00080000:3.0:1789762536.221266:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762536.221272:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762536.221278:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512960896:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789762536.221282:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762536.221302:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512961024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762536.221316:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762536.221320:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762536.221322:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762536.221328:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2510 00000100:00080000:3.0:1789762536.221332:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789762536.221336:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762536.221342:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762536.221349:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762536.221367:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512961152/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762536.221375:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762536/real 1789762536] req@ffff8dc0107af480 x1876699512960896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762536.221388:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762536.221392:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762536.221395:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762536.221403:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2510 00000100:00100000:1.0:1789762536.221408:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512960896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762536.221410:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762536.221419:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:1.0:1789762536.221426:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1789762536.221429:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1789762536.221432:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762536.221439:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512960896:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762536.221451:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512961280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762536.221463:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512961024:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762536.221492:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762536.221507:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762536/real 1789762536] req@ffff8dc0107ad180 x1876699512961024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762536.221521:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512961024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762536.221528:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762536.221531:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762536.221533:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512961024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762536.221538:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512961152:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762536.221553:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762536.221561:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762536/real 1789762536] req@ffff8dc0107aed80 x1876699512961152/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762536.221569:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512961152/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762536.221575:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762536.221577:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762536.221579:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512961152:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762536.221590:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512961280:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762536.221601:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762536.221608:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762536/real 1789762536] req@ffff8dc0107ad500 x1876699512961280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762536.221616:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512961280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762536.221621:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762536.221623:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762536.221625:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512961280:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762537.245024:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762538.269160:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762539.293143:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762540.317036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762541.341014:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762541.341029:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762541.341034:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1789762541.341037:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762541.341047:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2515 00000100:00080000:2.0:1789762541.341059:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762541.341068:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762541.341115:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512961408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762541.341145:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762541.341150:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762541.341153:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762541.341159:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2515 00000100:00080000:2.0:1789762541.341164:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762541.341169:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762541.341176:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512961408:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789762541.341180:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762541.341198:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512961536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762541.341211:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762541.341215:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762541.341217:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762541.341222:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2515 00000100:00080000:2.0:1789762541.341226:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789762541.341247:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762541.341247:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762541.341257:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762541.341272:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512961664/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762541.341285:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762541.341289:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762541.341292:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762541.341299:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2515 00000100:00100000:1.0:1789762541.341300:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762541/real 1789762541] req@ffff8dc00368df80 x1876699512961408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762541.341304:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762541.341311:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762541.341319:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762541.341326:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512961408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762541.341334:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512961792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762541.341339:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762541.341347:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762541.341352:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512961408:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762541.341375:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512961536:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762541.341393:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762541.341403:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762541/real 1789762541] req@ffff8dc00368e300 x1876699512961536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762541.341411:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512961536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762541.341418:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762541.341420:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762541.341422:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512961536:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762541.341425:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512961664:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762541.341437:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762541.341445:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762541/real 1789762541] req@ffff8dc00368ed80 x1876699512961664/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762541.341457:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512961664/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762541.341466:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762541.341469:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762541.341473:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512961664:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762541.341479:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512961792:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762541.341497:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762541.341505:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762541/real 1789762541] req@ffff8dc00368f800 x1876699512961792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762541.341513:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512961792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762541.341519:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762541.341521:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762541.341523:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512961792:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762542.365060:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762543.389003:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762544.413059:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762545.437059:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762546.461022:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762546.461033:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762546.461037:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762546.461048:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2521 00000100:00080000:2.0:1789762546.461062:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1789762546.461067:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762546.461075:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762546.461137:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512961920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762546.461163:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762546.461166:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762546.461168:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762546.461172:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2521 00000100:00080000:2.0:1789762546.461176:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762546.461179:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762546.461189:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762546.461199:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512962048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762546.461209:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512961920:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762546.461212:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762546.461229:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762546.461233:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762546.461241:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2521 00000100:00080000:2.0:1789762546.461246:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762546.461252:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762546.461261:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762546.461278:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512962176/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762546.461287:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762546.461290:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762546.461292:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762546.461295:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2521 00000100:00080000:2.0:1789762546.461298:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762546.461302:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762546.461309:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762546.461324:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512962304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762546.461330:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512962048:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762546.461375:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512962176:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762546.461396:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512962304:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762547.485158:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762548.509112:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762549.533300:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00010000:00010000:3.0:1789762549.533362:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762549.533363:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762549.533385:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762549.533434:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762549.533442:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762549.533447:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1789762549.533491:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762546/real 1789762549] req@ffff8dc00368ce00 x1876699512962304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762549.533524:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512962304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762549.533551:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762549.533559:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762549.533564:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512962304:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762549.533574:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762546/real 1789762549] req@ffff8dc00368c000 x1876699512962048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762549.533590:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512962048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762549.533599:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762549.533603:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762549.533606:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512962048:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762549.533616:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762546/real 1789762549] req@ffff8dc00368ea00 x1876699512962176/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762549.533630:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512962176/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762549.533638:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762549.533642:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762549.533645:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512962176:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762549.533652:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762546/real 1789762549] req@ffff8dc00368ca80 x1876699512961920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762549.533665:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512961920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762549.533681:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762549.533684:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762549.533688:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512961920:192.168.203.156@tcp:250:kworker.0 00010000:00010000:3.0:1789762550.557102:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762551.581001:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762551.581015:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762551.581021:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762551.581034:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2526 00000100:00080000:2.0:1789762551.581047:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762551.581057:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789762551.581069:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762551.581106:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512962432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762551.581140:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762551.581144:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762551.581146:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762551.581151:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2526 00000100:00080000:2.0:1789762551.581156:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762551.581162:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762551.581173:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762551.581188:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512962432:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762551.581190:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512962560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762551.581207:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762551.581211:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762551.581213:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762551.581218:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2526 00000100:00080000:2.0:1789762551.581221:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762551.581226:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762551.581233:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762551.581247:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512962688/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762551.581258:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762551.581263:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762551.581265:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762551.581269:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2526 00000100:00080000:2.0:1789762551.581273:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762551.581278:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1789762551.581285:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:2.0:1789762551.581286:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762551.581301:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512962816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762551.581339:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762551/real 1789762551] req@ffff8dc00368ca80 x1876699512962432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762551.581361:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512962432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762551.581378:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762551.581382:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762551.581386:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512962432:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762551.581414:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512962560:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762551.581441:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762551.581456:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762551/real 1789762551] req@ffff8dc00368ea00 x1876699512962560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762551.581469:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512962560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762551.581478:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762551.581482:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762551.581485:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512962560:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762551.581492:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512962688:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762551.581512:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762551.581523:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762551/real 1789762551] req@ffff8dc00368c000 x1876699512962688/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762551.581535:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512962688/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762551.581544:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762551.581547:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762551.581550:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512962688:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762551.581556:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512962816:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762551.581576:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762551.581588:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762551/real 1789762551] req@ffff8dc00368ce00 x1876699512962816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762551.581598:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512962816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762551.581605:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762551.581620:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762551.581625:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512962816:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762552.605193:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762553.629074:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762554.653038:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762555.677169:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762556.701042:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762556.701053:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1789762556.701055:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762556.701057:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762556.701069:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2531 00000100:00080000:3.0:1789762556.701081:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762556.701087:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762556.701130:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512962944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762556.701160:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762556.701171:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762556.701173:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762556.701177:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2531 00000100:00080000:3.0:1789762556.701182:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762556.701187:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762556.701200:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762556.701214:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512962944:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762556.701216:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512963072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762556.701232:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762556.701235:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762556.701237:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762556.701240:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2531 00000100:00080000:3.0:1789762556.701243:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762556.701246:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762556.701250:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762556.701259:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512963200/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762556.701266:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762556.701270:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762556.701271:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762556.701273:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2531 00000100:00080000:3.0:1789762556.701276:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762556.701278:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762556.701281:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762556.701288:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762556.701291:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512963328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762556.701331:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762556/real 1789762556] req@ffff8dc0107ac000 x1876699512962944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762556.701346:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512962944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762556.701354:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762556.701358:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762556.701361:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512962944:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762556.701380:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512963072:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762556.701395:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762556.701403:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762556/real 1789762556] req@ffff8dc0107ac700 x1876699512963072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762556.701411:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512963072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762556.701417:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762556.701419:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762556.701421:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512963072:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762556.701424:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512963200:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762556.701435:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762556.701441:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762556/real 1789762556] req@ffff8dc0107ae300 x1876699512963200/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762556.701449:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512963200/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762556.701454:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762556.701456:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762556.701458:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512963200:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762556.701461:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512963328:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762556.701471:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762556.701486:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762556/real 1789762556] req@ffff8dc0107aca80 x1876699512963328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762556.701494:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512963328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762556.701499:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762556.701501:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762556.701503:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512963328:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762557.725089:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762558.749051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762559.773045:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762560.797084:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762561.821061:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762561.821097:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762561.821106:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762561.821123:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2536 00000100:00080000:3.0:1789762561.821138:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762561.821148:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789762561.821194:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762561.821218:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699512963456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762561.821259:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762561.821265:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762561.821269:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762561.821282:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2536 00000100:00080000:3.0:1789762561.821290:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762561.821297:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762561.821309:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762561.821314:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512963456:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762561.821345:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699512963584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762561.821369:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762561.821377:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762561.821383:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762561.821394:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2536 00000100:00080000:3.0:1789762561.821408:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762561.821419:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1789762561.821423:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789762561.821444:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762561.821503:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762561/real 1789762561] req@ffff8dc0107adc00 x1876699512963456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762561.821527:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699512963456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762561.821540:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762561.821545:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762561.821549:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512963456:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762561.821577:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512963584:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762561.821600:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762561.821611:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762561/real 1789762561] req@ffff8dc0107afb80 x1876699512963584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762561.821621:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699512963584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762561.821628:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762561.821631:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762561.821634:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512963584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762561.821868:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512963712/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762561.821966:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762561.821968:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512963712:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1789762561.821988:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762561.821995:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762561.822009:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2536 00000800:00000100:2.0:1789762561.822018:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762561.822027:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762561.822036:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762561/real 1789762561] req@ffff8dc0107aca80 x1876699512963712/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762561.822046:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762561.822055:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512963712/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762561.822067:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1789762561.822067:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762561.822086:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762561.822090:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512963712:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789762561.822104:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512963840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762561.822159:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512963840:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762561.822184:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762561.822194:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762561/real 1789762561] req@ffff8dc0107ae300 x1876699512963840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762561.822204:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512963840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762561.822210:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762561.822213:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762561.822215:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512963840:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762562.845118:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762563.869030:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762564.893036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762565.917109:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762566.941047:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1789762566.941059:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762566.941062:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762566.941068:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762566.941083:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2541 00000100:00080000:3.0:1789762566.941097:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762566.941105:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762566.941157:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512963968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762566.941186:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762566.941190:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762566.941192:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762566.941196:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2541 00000100:00080000:3.0:1789762566.941202:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762566.941206:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762566.941219:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762566.941221:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512963968:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762566.941237:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512964096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762566.941250:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762566.941255:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762566.941257:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762566.941262:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2541 00000100:00080000:3.0:1789762566.941266:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762566.941271:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762566.941277:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762566.941291:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512964224/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762566.941302:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789762566.941304:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762566.941310:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762566.941313:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762566.941318:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2541 00000100:00080000:3.0:1789762566.941332:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762566.941336:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762566.941342:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762566.941354:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762566/real 1789762566] req@ffff8dc0107ac700 x1876699512963968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762566.941358:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512964352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762566.941380:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512963968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762566.941390:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762566.941394:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762566.941398:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512963968:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762566.941420:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512964096:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762566.941437:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762566.941446:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762566/real 1789762566] req@ffff8dc0107ac000 x1876699512964096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762566.941454:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512964096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762566.941460:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762566.941462:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762566.941465:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512964096:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762566.941468:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512964224:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762566.941479:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762566.941485:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762566/real 1789762566] req@ffff8dc0107ad500 x1876699512964224/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762566.941492:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512964224/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762566.941498:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762566.941499:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762566.941501:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512964224:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762566.941505:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512964352:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762566.941515:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762566.941521:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762566/real 1789762566] req@ffff8dc0107aed80 x1876699512964352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762566.941534:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512964352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762566.941539:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762566.941541:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762566.941543:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512964352:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762567.965071:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762568.989065:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762570.013123:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762571.037143:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762572.061001:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762572.061010:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762572.061013:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762572.061023:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2546 00000100:00080000:0.0:1789762572.061032:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762572.061038:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762572.061071:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148a80 x1876699512964480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762572.061096:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762572.061100:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762572.061102:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762572.061106:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2546 00000100:00080000:0.0:1789762572.061110:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762572.061114:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762572.061123:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762572.061140:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514aa00 x1876699512964608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762572.061151:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512964480:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789762572.061155:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762572.061161:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762572.061163:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762572.061168:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2546 00000100:00080000:0.0:1789762572.061172:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762572.061177:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762572.061183:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762572.061199:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148700 x1876699512964736/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762572.061214:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1789762572.061222:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762572.061226:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762572.061227:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762572.061232:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2546 00000100:00080000:0.0:1789762572.061235:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762572.061239:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762572.061244:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762572.061254:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762572/real 1789762572] req@ffff8dc005148a80 x1876699512964480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762572.061256:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514ad80 x1876699512964864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762572.061296:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148a80 x1876699512964480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762572.061306:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762572.061309:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762572.061313:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512964480:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762572.061333:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512964608:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762572.061351:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.0:1789762572.061352:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762572.061364:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762572/real 1789762572] req@ffff8dc00514aa00 x1876699512964608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762572.061378:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514aa00 x1876699512964608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762572.061384:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762572.061386:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762572.061389:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512964608:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762572.061393:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512964736:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762572.061405:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762572.061414:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762572/real 1789762572] req@ffff8dc005148700 x1876699512964736/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762572.061423:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148700 x1876699512964736/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762572.061428:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762572.061430:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762572.061432:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512964736:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762572.061436:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512964864:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762572.061445:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762572.061451:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762572/real 1789762572] req@ffff8dc00514ad80 x1876699512964864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762572.061458:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514ad80 x1876699512964864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762572.061467:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762572.061469:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762572.061472:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512964864:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762573.085111:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762574.109136:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762575.133156:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762576.157114:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762577.181000:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762577.181008:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762577.181012:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762577.181021:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2551 00000100:00080000:2.0:1789762577.181030:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762577.181036:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762577.181070:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512964992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1789762577.181076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762577.181102:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762577.181108:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762577.181111:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762577.181117:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2551 00000100:00080000:2.0:1789762577.181125:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762577.181130:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762577.181140:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762577.181149:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512964992:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762577.181159:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512965120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762577.181173:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762577.181178:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762577.181180:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762577.181184:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2551 00000100:00080000:2.0:1789762577.181189:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762577.181193:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762577.181199:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762577.181212:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512965248/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762577.181222:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762577.181229:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762577.181233:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762577.181235:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762577.181241:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2551 00000100:00080000:2.0:1789762577.181245:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762577.181250:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762577.181255:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762577.181260:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762577/real 1789762577] req@ffff8dc00368f100 x1876699512964992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762577.181271:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512965376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762577.181278:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512964992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762577.181293:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762577.181296:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762577.181300:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512964992:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762577.181328:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512965120:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762577.181344:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762577.181353:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762577/real 1789762577] req@ffff8dc00368f480 x1876699512965120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762577.181362:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512965120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762577.181368:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762577.181370:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762577.181372:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512965120:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762577.181376:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512965248:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762577.181388:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762577.181394:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762577/real 1789762577] req@ffff8dc00368fb80 x1876699512965248/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762577.181402:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512965248/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762577.181407:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762577.181409:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762577.181411:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512965248:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762577.181414:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512965376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762577.181424:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762577.181430:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762577/real 1789762577] req@ffff8dc00368d880 x1876699512965376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762577.181438:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512965376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762577.181443:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762577.181445:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762577.181446:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512965376:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762578.205122:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762579.229048:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762580.253137:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762581.277058:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762582.301019:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762582.301027:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762582.301031:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762582.301041:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2556 00000100:00080000:1.0:1789762582.301050:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762582.301056:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789762582.301063:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789762582.301098:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512965504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762582.301129:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762582.301134:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762582.301136:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762582.301142:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2556 00000100:00080000:1.0:1789762582.301147:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762582.301154:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512965504:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789762582.301155:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762582.301178:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762582.301196:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512965632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762582.301209:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762582.301212:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762582.301216:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789762582.301217:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789762582.301225:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2556 00000100:00080000:1.0:1789762582.301229:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762582.301233:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762582.301237:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762582.301249:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512965760/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762582.301257:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762582/real 1789762582] req@ffff8dc005c97800 x1876699512965504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762582.301258:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762582.301266:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762582.301268:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762582.301277:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2556 00000100:00080000:1.0:1789762582.301284:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762582.301288:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512965504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762582.301291:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762582.301302:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762582.301305:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762582.301310:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762582.301313:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512965504:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762582.301319:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512965888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762582.301336:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512965632:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762582.301354:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762582.301363:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762582/real 1789762582] req@ffff8dc005c95c00 x1876699512965632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762582.301371:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512965632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762582.301376:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762582.301378:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762582.301380:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512965632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762582.301384:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512965760:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762582.301395:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762582.301401:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762582/real 1789762582] req@ffff8dc005c94a80 x1876699512965760/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762582.301408:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512965760/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762582.301418:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762582.301420:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762582.301422:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512965760:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762582.301425:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512965888:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762582.301435:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762582.301440:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762582/real 1789762582] req@ffff8dc005c97100 x1876699512965888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762582.301447:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512965888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762582.301453:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762582.301454:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762582.301456:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512965888:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762583.325153:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762584.349025:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762585.373083:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762586.397041:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762587.421002:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762587.421011:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762587.421015:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762587.421024:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2562 00010000:00010000:0.0:1789762587.421026:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762587.421033:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762587.421041:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762587.421079:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512966016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762587.421099:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762587.421102:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762587.421103:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762587.421106:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2562 00000100:00080000:1.0:1789762587.421108:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762587.421112:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762587.421120:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762587.421131:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512966144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762587.421139:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762587.421142:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1789762587.421142:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512966016:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789762587.421146:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762587.421152:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2562 00000100:00080000:1.0:1789762587.421154:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762587.421156:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762587.421160:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762587.421172:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512966272/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762587.421183:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762587.421187:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762587.421189:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762587.421194:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2562 00000100:00080000:1.0:1789762587.421198:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762587.421205:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1789762587.421210:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1789762587.421213:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762587.421228:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512966400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762587.421254:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762587/real 1789762587] req@ffff8dc005c95180 x1876699512966016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762587.421269:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512966016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762587.421278:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762587.421280:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762587.421284:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512966016:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762587.421302:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512966144:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762587.421320:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762587.421328:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762587/real 1789762587] req@ffff8dc005c97100 x1876699512966144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762587.421340:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512966144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762587.421349:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762587.421353:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762587.421356:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512966144:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762587.421363:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512966272:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762587.421385:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762587.421395:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762587/real 1789762587] req@ffff8dc005c94a80 x1876699512966272/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762587.421404:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512966272/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762587.421410:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762587.421412:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762587.421414:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512966272:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762587.421418:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512966400:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762587.421431:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762587.421438:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762587/real 1789762587] req@ffff8dc005c95c00 x1876699512966400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762587.421445:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512966400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762587.421450:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762587.421452:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762587.421454:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512966400:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762588.444995:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762589.469073:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762590.493043:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762591.517100:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762592.540990:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762592.540999:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762592.541003:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762592.541020:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2567 00000100:00080000:2.0:1789762592.541028:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762592.541033:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789762592.541058:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762592.541068:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512966528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762592.541090:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762592.541093:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762592.541094:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762592.541097:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2567 00000100:00080000:2.0:1789762592.541100:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762592.541103:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762592.541111:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762592.541125:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512966528:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762592.541125:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512966656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762592.541143:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762592.541147:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762592.541149:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762592.541153:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2567 00000100:00080000:2.0:1789762592.541158:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762592.541163:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762592.541169:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789762592.541188:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762592.541193:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512966784/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762592.541204:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762592.541206:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762592.541207:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762592.541210:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2567 00000100:00080000:2.0:1789762592.541215:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762592.541218:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762592.541222:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762592/real 1789762592] req@ffff8dc00368d880 x1876699512966528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789762592.541225:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762592.541245:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512966528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762592.541254:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512966912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762592.541256:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762592.541264:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762592.541272:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512966528:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762592.541296:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512966656:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762592.541312:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762592.541320:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762592/real 1789762592] req@ffff8dc00368fb80 x1876699512966656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762592.541328:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512966656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762592.541334:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762592.541337:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762592.541339:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512966656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762592.541342:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512966784:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762592.541353:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762592.541359:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762592/real 1789762592] req@ffff8dc00368f480 x1876699512966784/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762592.541371:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512966784/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762592.541377:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762592.541378:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762592.541380:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512966784:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762592.541383:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512966912:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762592.541393:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762592.541399:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762592/real 1789762592] req@ffff8dc00368f100 x1876699512966912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762592.541407:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512966912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762592.541411:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762592.541413:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762592.541415:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512966912:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762593.565042:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762594.589039:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762595.613061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762596.637042:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762597.661010:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762597.661021:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762597.661027:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762597.661041:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2572 00010000:00010000:3.0:1789762597.661043:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762597.661052:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762597.661060:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762597.661104:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512967040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762597.661130:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762597.661133:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762597.661135:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762597.661139:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2572 00000100:00080000:2.0:1789762597.661143:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762597.661146:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762597.661155:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512967040:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789762597.661156:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762597.661173:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512967168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762597.661183:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762597.661186:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762597.661188:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762597.661191:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2572 00000100:00080000:2.0:1789762597.661194:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762597.661196:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762597.661199:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762597.661207:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512967296/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762597.661219:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762597.661222:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762597.661223:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762597.661225:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2572 00000100:00080000:2.0:1789762597.661228:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789762597.661232:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762597.661236:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762597.661240:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762597.661248:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512967424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762597.661274:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762597/real 1789762597] req@ffff8dc00368ce00 x1876699512967040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762597.661290:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512967040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762597.661304:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762597.661308:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762597.661311:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512967040:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762597.661343:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512967168:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762597.661359:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762597.661368:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762597/real 1789762597] req@ffff8dc00368c000 x1876699512967168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762597.661376:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512967168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762597.661381:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762597.661383:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762597.661386:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512967168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762597.661398:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512967296:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762597.661412:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762597.661419:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762597/real 1789762597] req@ffff8dc00368ea00 x1876699512967296/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762597.661426:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512967296/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762597.661431:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762597.661433:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762597.661436:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512967296:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762597.661439:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512967424:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762597.661459:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762597.661466:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762597/real 1789762597] req@ffff8dc00368ca80 x1876699512967424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762597.661493:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512967424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762597.661499:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762597.661501:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762597.661503:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512967424:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762598.685079:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762599.709025:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762600.733075:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762601.757095:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762602.781017:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1789762602.781018:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762602.781029:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762602.781033:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762602.781045:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2577 00000100:00080000:2.0:1789762602.781056:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762602.781065:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762602.781109:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512967552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762602.781132:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762602.781134:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762602.781136:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762602.781139:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2577 00000100:00080000:2.0:1789762602.781141:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762602.781145:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762602.781154:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762602.781163:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512967680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762602.781169:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762602.781172:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762602.781173:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762602.781176:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2577 00000100:00080000:2.0:1789762602.781178:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762602.781181:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762602.781185:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762602.781193:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512967808/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762602.781201:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762602.781207:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1789762602.781208:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512967552:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762602.781210:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762602.781225:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2577 00000100:00080000:2.0:1789762602.781229:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762602.781232:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762602.781236:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762602.781243:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512967936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762602.781302:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762602.781342:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762602/real 1789762602] req@ffff8dc00368f800 x1876699512967552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762602.781358:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512967552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762602.781367:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762602.781371:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762602.781374:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512967552:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762602.781381:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512967680:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762602.781399:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762602.781409:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762602/real 1789762602] req@ffff8dc00368ed80 x1876699512967680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762602.781418:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512967680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762602.781424:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762602.781427:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762602.781431:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512967680:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762602.781466:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512967808:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762602.781491:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762602.781516:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762602/real 1789762602] req@ffff8dc00368e300 x1876699512967808/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762602.781526:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512967808/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762602.781533:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762602.781535:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762602.781537:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512967808:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762602.781543:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512967936:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762602.781558:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762602.781565:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762602/real 1789762602] req@ffff8dc00368df80 x1876699512967936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762602.781573:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512967936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762602.781579:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762602.781581:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762602.781583:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512967936:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762603.805027:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762604.829072:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762605.853013:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762606.877044:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762607.901020:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762607.901029:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762607.901033:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1789762607.901038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762607.901043:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2582 00000100:00080000:1.0:1789762607.901052:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762607.901058:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762607.901090:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512968064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762607.901114:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762607.901119:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762607.901121:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762607.901126:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2582 00000100:00080000:1.0:1789762607.901131:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762607.901138:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762607.901138:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512968064:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1789762607.901154:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762607.901170:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512968192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762607.901180:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762607.901183:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762607.901185:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762607.901187:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2582 00000100:00080000:1.0:1789762607.901192:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762607.901194:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762607.901198:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:2.0:1789762607.901204:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762607.901208:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512968320/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762607.901217:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762607.901220:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762607.901222:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762607.901225:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2582 00000100:00080000:1.0:1789762607.901227:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762607.901230:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762607.901233:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762607.901241:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512968448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762607.901248:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762607/real 1789762607] req@ffff8dc005c97800 x1876699512968064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762607.901264:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512968064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762607.901273:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762607.901276:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762607.901279:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512968064:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762607.901298:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512968192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762607.901313:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762607.901322:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762607/real 1789762607] req@ffff8dc005c96a00 x1876699512968192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762607.901331:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512968192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762607.901340:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762607.901343:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762607.901346:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512968192:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762607.901353:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512968320:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762607.901374:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762607.901386:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762607/real 1789762607] req@ffff8dc005c94700 x1876699512968320/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762607.901405:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512968320/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762607.901414:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762607.901418:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762607.901421:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512968320:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762607.901427:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512968448:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762607.901450:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762607.901478:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762607/real 1789762607] req@ffff8dc005c95f80 x1876699512968448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762607.901492:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512968448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762607.901501:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762607.901504:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762607.901508:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512968448:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762608.925077:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762609.949064:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762610.973078:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762611.997022:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762613.020999:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762613.021010:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762613.021014:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762613.021025:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2587 00000100:00080000:1.0:1789762613.021036:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762613.021044:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789762613.021044:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789762613.021092:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512968576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762613.021118:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762613.021120:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762613.021122:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762613.021125:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2587 00000100:00080000:1.0:1789762613.021127:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762613.021131:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762613.021141:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762613.021151:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512968704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762613.021160:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762613.021168:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1789762613.021169:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512968576:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789762613.021171:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762613.021184:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2587 00000100:00080000:1.0:1789762613.021188:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762613.021190:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762613.021194:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762613.021204:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512968832/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762613.021211:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762613.021214:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762613.021215:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762613.021218:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2587 00000100:00080000:1.0:1789762613.021222:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762613.021225:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762613.021230:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762613.021238:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512968960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762613.021285:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512968704:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762613.021313:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512968832:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762613.021341:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512968960:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762614.045104:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762615.069045:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762616.093101:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789762616.093221:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1789762616.093272:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789762616.093290:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789762616.093342:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789762616.093354:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789762616.093367:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1789762616.093413:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762613/real 1789762616] req@ffff8dc005c94380 x1876699512968704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762616.093456:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512968704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762616.093473:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762616.093479:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762616.093485:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512968704:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762616.093495:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762613/real 1789762616] req@ffff8dc005c97b80 x1876699512968832/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762616.093510:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512968832/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762616.093519:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762616.093524:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762616.093528:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512968832:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762616.093537:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762613/real 1789762616] req@ffff8dc005c96300 x1876699512968960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762616.093550:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512968960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762616.093560:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762616.093566:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762616.093570:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512968960:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762616.093577:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762613/real 1789762616] req@ffff8dc005c95500 x1876699512968576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762616.093591:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512968576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762616.093601:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762616.093604:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762616.093608:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512968576:192.168.203.156@tcp:250:kworker.0 00010000:00010000:2.0:1789762617.117085:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762618.140972:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762618.140980:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762618.140985:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762618.140994:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2592 00000100:00080000:1.0:1789762618.141003:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762618.141008:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789762618.141011:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789762618.141042:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512969088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762618.141064:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762618.141066:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762618.141067:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762618.141070:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2592 00000100:00080000:1.0:1789762618.141072:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762618.141075:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762618.141083:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762618.141092:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512969216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762618.141100:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762618.141109:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762618.141110:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762618.141115:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2592 00000100:00080000:1.0:1789762618.141119:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762618.141126:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762618.141127:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512969088:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1789762618.141131:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762618.141152:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512969344/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762618.141167:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762618.141171:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762618.141173:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762618.141177:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2592 00000100:00080000:1.0:1789762618.141182:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762618.141188:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762618.141193:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762618.141208:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512969472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762618.141229:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762618.141284:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762618/real 1789762618] req@ffff8dc005c94e00 x1876699512969088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762618.141307:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512969088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762618.141320:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762618.141326:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762618.141331:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512969088:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762618.141347:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512969216:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762618.141374:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762618.141405:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762618/real 1789762618] req@ffff8dc005c97480 x1876699512969216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762618.141421:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512969216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762618.141431:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762618.141435:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762618.141439:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512969216:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762618.141471:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512969344:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762618.141495:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762618.141508:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762618/real 1789762618] req@ffff8dc005c96680 x1876699512969344/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762618.141526:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512969344/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762618.141535:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762618.141539:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762618.141543:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512969344:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762618.141550:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512969472:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762618.141567:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762618.141578:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762618/real 1789762618] req@ffff8dc005c95880 x1876699512969472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762618.141590:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512969472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762618.141606:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762618.141610:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762618.141613:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512969472:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762619.165124:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762620.189029:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762621.213003:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762622.237091:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762623.260992:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762623.261000:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762623.261003:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762623.261013:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2597 00000100:00080000:1.0:1789762623.261021:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762623.261027:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789762623.261054:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789762623.261062:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96d80 x1876699512969600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762623.261084:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762623.261087:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762623.261088:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762623.261091:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2597 00000100:00080000:1.0:1789762623.261093:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762623.261096:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762623.261104:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762623.261115:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512969600:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762623.261116:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512969728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762623.261129:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762623.261131:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762623.261132:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762623.261134:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2597 00000100:00080000:1.0:1789762623.261137:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762623.261139:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762623.261143:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762623.261152:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512969856/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762623.261159:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762623.261161:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762623.261162:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762623.261165:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2597 00000100:00080000:1.0:1789762623.261169:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762623.261171:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762623.261174:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762623.261179:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762623.261185:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512969984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762623.261218:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762623/real 1789762623] req@ffff8dc005c96d80 x1876699512969600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762623.261232:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96d80 x1876699512969600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762623.261240:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762623.261243:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762623.261246:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512969600:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762623.261264:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512969728:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762623.261279:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762623.261287:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762623/real 1789762623] req@ffff8dc005c95880 x1876699512969728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762623.261294:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512969728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762623.261304:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762623.261306:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762623.261309:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512969728:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762623.261312:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512969856:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762623.261323:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762623.261329:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762623/real 1789762623] req@ffff8dc005c96680 x1876699512969856/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762623.261336:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512969856/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762623.261342:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762623.261358:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762623.261362:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512969856:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762623.261368:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512969984:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762623.261389:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762623.261401:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762623/real 1789762623] req@ffff8dc005c97480 x1876699512969984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762623.261415:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512969984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762623.261420:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762623.261422:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762623.261425:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512969984:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762624.285027:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762625.309047:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762626.333133:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762627.357063:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762628.381047:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762628.381058:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762628.381063:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762628.381076:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2602 00000100:00080000:0.0:1789762628.381088:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762628.381097:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762628.381160:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514bb80 x1876699512970112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762628.381190:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762628.381193:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762628.381195:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762628.381198:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2602 00000100:00080000:0.0:1789762628.381201:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762628.381205:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762628.381216:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762628.381228:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b480 x1876699512970240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762628.381236:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762628.381238:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512970112:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789762628.381242:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762628.381258:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762628.381263:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2602 00000100:00080000:0.0:1789762628.381267:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762628.381272:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762628.381278:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762628.381294:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148000 x1876699512970368/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762628.381327:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762628.381330:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762628.381331:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762628.381335:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2602 00000100:00080000:0.0:1789762628.381339:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789762628.381342:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1789762628.381346:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762628.381352:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762628.381362:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148380 x1876699512970496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762628.381397:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762628/real 1789762628] req@ffff8dc00514bb80 x1876699512970112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762628.381416:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514bb80 x1876699512970112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762628.381426:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762628.381431:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762628.381435:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512970112:192.168.203.156@tcp:250:kworker.0 00010000:00010000:3.0:1789762628.381454:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762628.381462:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512970240:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762628.381492:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762628.381501:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762628/real 1789762628] req@ffff8dc00514b480 x1876699512970240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762628.381510:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b480 x1876699512970240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762628.381516:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762628.381518:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762628.381521:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512970240:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762628.381524:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512970368:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762628.381538:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762628.381544:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762628/real 1789762628] req@ffff8dc005148000 x1876699512970368/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762628.381552:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148000 x1876699512970368/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762628.381558:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762628.381559:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762628.381561:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512970368:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762628.381576:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512970496:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762628.381587:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762628.381593:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762628/real 1789762628] req@ffff8dc005148380 x1876699512970496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762628.381601:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148380 x1876699512970496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762628.381606:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762628.381608:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762628.381610:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512970496:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762629.405098:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762630.428999:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762631.453013:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762632.477106:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762633.501051:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762633.501063:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762633.501069:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762633.501073:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762633.501082:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2608 00000100:00080000:1.0:1789762633.501090:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762633.501096:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762633.501127:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512970624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762633.501148:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762633.501152:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762633.501153:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762633.501156:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2608 00000100:00080000:1.0:1789762633.501159:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762633.501163:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762633.501170:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762633.501183:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512970624:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762633.501184:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512970752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762633.501202:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762633.501205:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762633.501206:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762633.501209:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2608 00000100:00080000:1.0:1789762633.501212:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762633.501214:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762633.501218:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762633.501228:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512970880/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762633.501236:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762633.501239:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762633.501240:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789762633.501245:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789762633.501247:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2608 00000100:00080000:1.0:1789762633.501251:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762633.501255:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762633.501259:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762633.501280:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512971008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762633.501298:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762633/real 1789762633] req@ffff8dc005c94e00 x1876699512970624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762633.501318:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512970624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762633.501326:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762633.501329:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762633.501332:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512970624:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762633.501350:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512970752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762633.501367:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762633.501375:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762633/real 1789762633] req@ffff8dc005c95500 x1876699512970752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762633.501383:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512970752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762633.501388:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762633.501391:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762633.501393:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512970752:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762633.501396:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512970880:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762633.501407:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762633.501413:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762633/real 1789762633] req@ffff8dc005c96300 x1876699512970880/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762633.501421:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512970880/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762633.501430:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762633.501433:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762633.501435:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512970880:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762633.501438:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512971008:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762633.501449:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762633.501455:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762633/real 1789762633] req@ffff8dc005c97b80 x1876699512971008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762633.501469:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512971008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762633.501478:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762633.501482:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762633.501486:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512971008:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762634.525173:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762635.549200:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762636.573044:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762637.597019:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762638.621074:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762638.621103:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762638.621112:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762638.621117:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762638.621128:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2613 00000100:00080000:2.0:1789762638.621140:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762638.621148:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762638.621199:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512971136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762638.621256:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762638.621264:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762638.621268:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762638.621275:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2613 00000100:00080000:2.0:1789762638.621281:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762638.621287:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762638.621300:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512971136:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789762638.621307:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762638.621328:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512971264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762638.621342:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762638.621346:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762638.621348:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762638.621351:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2613 00000100:00080000:2.0:1789762638.621355:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762638.621359:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762638.621364:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762638.621376:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512971392/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762638.621387:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762638.621394:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1789762638.621395:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762638.621397:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762638.621408:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2613 00000100:00080000:2.0:1789762638.621414:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762638.621420:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762638.621426:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762638.621438:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512971520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762638.621455:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762638/real 1789762638] req@ffff8dc00368d180 x1876699512971136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762638.621480:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512971136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762638.621493:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762638.621497:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762638.621502:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512971136:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762638.621527:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512971264:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762638.621549:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762638.621562:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762638/real 1789762638] req@ffff8dc00368c700 x1876699512971264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762638.621575:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512971264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762638.621584:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762638.621587:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762638.621591:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512971264:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762638.621596:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512971392:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762638.621612:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762638.621621:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762638/real 1789762638] req@ffff8dc00368dc00 x1876699512971392/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762638.621632:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512971392/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762638.621648:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762638.621651:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762638.621655:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512971392:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762638.621660:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512971520:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762638.621675:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762638.621684:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762638/real 1789762638] req@ffff8dc00368e680 x1876699512971520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762638.621695:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512971520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762638.621704:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762638.621706:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762638.621709:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512971520:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762639.645152:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762640.669038:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762641.693033:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762642.717074:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762643.741097:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762643.741147:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762643.741157:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762643.741163:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762643.741175:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2618 00000100:00080000:1.0:1789762643.741216:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762643.741222:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762643.741257:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512971648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762643.741281:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762643.741285:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762643.741286:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762643.741289:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2618 00000100:00080000:1.0:1789762643.741292:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762643.741295:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762643.741303:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762643.741317:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512971776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762643.741317:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512971648:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789762643.741332:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762643.741334:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762643.741336:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762643.741340:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2618 00000100:00080000:1.0:1789762643.741342:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762643.741344:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762643.741347:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762643.741355:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512971904/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762643.741362:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762643.741365:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762643.741366:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762643.741369:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2618 00000100:00080000:1.0:1789762643.741372:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762643.741375:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1789762643.741379:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1789762643.741382:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762643.741392:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512972032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762643.741436:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762643/real 1789762643] req@ffff8dc005c94380 x1876699512971648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762643.741450:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512971648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762643.741457:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762643.741460:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762643.741463:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512971648:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762643.741502:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512971776:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762643.741527:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762643.741541:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762643/real 1789762643] req@ffff8dc005c95f80 x1876699512971776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762643.741553:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512971776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762643.741560:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762643.741563:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762643.741567:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512971776:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762643.741572:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512971904:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762643.741588:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762643.741596:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762643/real 1789762643] req@ffff8dc005c94700 x1876699512971904/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762643.741607:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512971904/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762643.741614:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762643.741617:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762643.741619:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512971904:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762643.741634:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512972032:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762643.741650:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762643.741659:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762643/real 1789762643] req@ffff8dc005c96a00 x1876699512972032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762668 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762643.741669:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512972032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762668 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762643.741677:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762643.741680:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762643.741682:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512972032:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762644.765137:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762645.789109:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762646.813165:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762647.837108:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762648.860982:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762648.860994:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762648.860998:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762648.861009:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2623 00000100:00080000:0.0:1789762648.861021:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762648.861028:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762648.861072:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149c00 x1876699512972160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762648.861097:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762648.861101:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762648.861102:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762648.861105:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2623 00000100:00080000:0.0:1789762648.861108:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762648.861110:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762648.861119:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762648.861128:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149880 x1876699512972288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762648.861135:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762648.861138:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762648.861139:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762648.861142:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2623 00000100:00080000:0.0:1789762648.861143:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762648.861145:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762648.861148:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762648.861170:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149180 x1876699512972416/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762648.861178:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762648.861181:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762648.861183:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762648.861186:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2623 00000100:00080000:0.0:1789762648.861188:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762648.861191:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762648.861195:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762648.861202:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149f80 x1876699512972544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762648.861215:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512972160:192.168.203.156@tcp:250:kworker.0 00010000:00010000:0.0:1789762648.861250:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.0:1789762648.861299:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762648.861349:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762648/real 1789762648] req@ffff8dc005149c00 x1876699512972160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762648.861372:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149c00 x1876699512972160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762648.861381:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762648.861385:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762648.861388:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512972160:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762648.861395:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512972288:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762648.861415:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762648.861425:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762648/real 1789762648] req@ffff8dc005149880 x1876699512972288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762648.861434:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149880 x1876699512972288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762648.861440:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762648.861443:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762648.861446:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512972288:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762648.861459:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512972416:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762648.861477:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762648.861485:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762648/real 1789762648] req@ffff8dc005149180 x1876699512972416/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762648.861494:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149180 x1876699512972416/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762648.861505:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762648.861507:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762648.861510:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512972416:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762648.861532:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512972544:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762648.861545:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762648.861553:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762648/real 1789762648] req@ffff8dc005149f80 x1876699512972544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762648.861561:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149f80 x1876699512972544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762648.861567:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762648.861569:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762648.861571:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512972544:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762649.885103:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762650.908994:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762651.933030:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762652.957063:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762653.981030:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762653.981044:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762653.981054:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762653.981058:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762653.981069:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2628 00000100:00080000:0.0:1789762653.981080:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762653.981086:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762653.981134:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b800 x1876699512972672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762653.981154:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762653.981157:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762653.981158:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762653.981162:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2628 00000100:00080000:0.0:1789762653.981165:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762653.981169:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762653.981177:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762653.981183:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512972672:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762653.981189:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514a300 x1876699512972800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762653.981198:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762653.981201:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762653.981202:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762653.981205:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2628 00000100:00080000:0.0:1789762653.981207:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762653.981209:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762653.981214:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762653.981222:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148e00 x1876699512972928/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762653.981229:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762653.981232:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762653.981233:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762653.981236:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2628 00000100:00080000:0.0:1789762653.981240:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789762653.981240:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1789762653.981247:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762653.981251:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762653.981259:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b100 x1876699512973056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762653.981281:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762653/real 1789762653] req@ffff8dc00514b800 x1876699512972672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762653.981295:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b800 x1876699512972672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762653.981303:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762653.981306:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762653.981309:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512972672:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762653.981327:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512972800:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762653.981341:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762653.981349:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762653/real 1789762653] req@ffff8dc00514a300 x1876699512972800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762653.981357:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514a300 x1876699512972800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762653.981362:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762653.981364:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762653.981366:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512972800:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762653.981369:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512972928:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762653.981381:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762653.981387:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762653/real 1789762653] req@ffff8dc005148e00 x1876699512972928/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762653.981394:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148e00 x1876699512972928/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762653.981399:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762653.981401:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762653.981403:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512972928:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762653.981407:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512973056:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762653.981417:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762653.981423:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762653/real 1789762653] req@ffff8dc00514b100 x1876699512973056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762653.981430:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b100 x1876699512973056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762653.981435:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762653.981436:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762653.981438:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512973056:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762655.005092:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762656.029127:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762657.053085:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762658.077125:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762659.101025:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762659.101035:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762659.101041:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1789762659.101052:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762659.101054:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2633 00000100:00080000:1.0:1789762659.101068:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762659.101090:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762659.101128:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512973184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762659.101161:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762659.101165:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762659.101166:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762659.101170:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2633 00000100:00080000:1.0:1789762659.101173:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762659.101177:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762659.101184:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762659.101197:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512973312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762659.101207:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762659.101207:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512973184:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789762659.101217:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762659.101219:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762659.101223:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2633 00000100:00080000:1.0:1789762659.101225:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762659.101228:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762659.101231:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762659.101240:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512973440/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762659.101248:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762659.101251:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762659.101252:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762659.101254:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2633 00000100:00080000:1.0:1789762659.101257:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762659.101260:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762659.101263:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762659.101272:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762659.101273:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512973568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762659.101312:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762659/real 1789762659] req@ffff8dc005c97800 x1876699512973184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762659.101326:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512973184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762659.101334:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762659.101337:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762659.101340:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512973184:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762659.101359:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512973312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762659.101374:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762659.101382:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762659/real 1789762659] req@ffff8dc005c95c00 x1876699512973312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762659.101396:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512973312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762659.101401:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762659.101403:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762659.101405:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512973312:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762659.101409:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512973440:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762659.101421:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762659.101427:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762659/real 1789762659] req@ffff8dc005c94a80 x1876699512973440/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762659.101435:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512973440/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762659.101440:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762659.101442:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762659.101444:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512973440:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762659.101448:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512973568:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762659.101458:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762659.101464:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762659/real 1789762659] req@ffff8dc005c97100 x1876699512973568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762659.101471:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512973568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762659.101481:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762659.101483:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762659.101485:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512973568:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762660.125109:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762661.149014:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762662.173003:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762663.197017:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762664.220995:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762664.221033:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762664.221054:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762664.221059:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762664.221070:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2638 00000100:00080000:0.0:1789762664.221081:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762664.221088:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762664.221126:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149500 x1876699512973696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762664.221148:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762664.221151:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762664.221152:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762664.221155:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2638 00000100:00080000:0.0:1789762664.221159:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762664.221162:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762664.221169:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762664.221180:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b100 x1876699512973824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762664.221188:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762664.221191:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512973696:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789762664.221192:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762664.221196:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762664.221203:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2638 00000100:00080000:0.0:1789762664.221206:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762664.221208:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762664.221211:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762664.221221:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148e00 x1876699512973952/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762664.221234:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762664.221237:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762664.221238:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762664.221240:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2638 00000100:00080000:0.0:1789762664.221243:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762664.221245:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762664.221248:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762664.221253:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762664.221258:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514a300 x1876699512974080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762664.221288:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762664/real 1789762664] req@ffff8dc005149500 x1876699512973696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762664.221301:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149500 x1876699512973696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762664.221309:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762664.221312:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762664.221315:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512973696:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762664.221332:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512973824:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762664.221346:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762664.221354:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762664/real 1789762664] req@ffff8dc00514b100 x1876699512973824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762664.221361:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b100 x1876699512973824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762664.221367:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762664.221370:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762664.221372:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512973824:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762664.221376:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512973952:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762664.221386:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762664.221392:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762664/real 1789762664] req@ffff8dc005148e00 x1876699512973952/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762664.221399:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148e00 x1876699512973952/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762664.221404:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762664.221406:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762664.221408:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512973952:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762664.221411:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512974080:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762664.221425:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762664.221435:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762664/real 1789762664] req@ffff8dc00514a300 x1876699512974080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762664.221447:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514a300 x1876699512974080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762664.221456:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762664.221460:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762664.221463:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512974080:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762665.245011:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762666.269018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762667.293042:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762668.317177:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762669.341082:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1789762669.341084:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762669.341101:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762669.341109:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762669.341128:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2643 00000100:00080000:2.0:1789762669.341149:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762669.341162:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762669.341209:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512974208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762669.341254:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762669.341261:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762669.341264:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762669.341292:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2643 00000100:00080000:2.0:1789762669.341299:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762669.341308:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762669.341322:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762669.341335:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512974208:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762669.341345:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512974336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762669.341362:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762669.341366:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762669.341369:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762669.341373:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2643 00000100:00080000:2.0:1789762669.341378:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762669.341381:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762669.341387:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762669.341401:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512974464/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762669.341414:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762669.341419:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1789762669.341423:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762669.341424:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762669.341435:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2643 00000100:00080000:2.0:1789762669.341440:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762669.341445:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762669.341451:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762669.341464:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512974592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762669.341490:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762669/real 1789762669] req@ffff8dc00368e680 x1876699512974208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762669.341517:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512974208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762669.341531:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762669.341536:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762669.341541:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512974208:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762669.341574:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512974336:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762669.341608:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762669.341624:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762669/real 1789762669] req@ffff8dc00368dc00 x1876699512974336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762669.341641:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512974336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762669.341652:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762669.341659:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762669.341678:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512974336:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762669.341685:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512974464:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762669.341709:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762669.341731:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762669/real 1789762669] req@ffff8dc00368c700 x1876699512974464/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762669.341757:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512974464/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762669.341770:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762669.341776:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762669.341780:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512974464:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762669.341787:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512974592:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762669.341805:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762669.341816:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762669/real 1789762669] req@ffff8dc00368d180 x1876699512974592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762669.341829:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512974592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762669.341837:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762669.341840:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762669.341843:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512974592:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762670.365029:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762671.389071:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762672.413063:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762673.437052:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762674.460994:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762674.461002:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762674.461006:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762674.461016:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2649 00000100:00080000:2.0:1789762674.461024:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762674.461030:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762674.461063:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512974720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1789762674.461068:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762674.461083:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762674.461087:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762674.461089:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762674.461092:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2649 00000100:00080000:2.0:1789762674.461095:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762674.461098:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762674.461103:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512974720:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789762674.461109:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762674.461121:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512974848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762674.461129:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762674.461132:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762674.461134:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762674.461136:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2649 00000100:00080000:2.0:1789762674.461139:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762674.461141:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762674.461144:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762674.461152:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512974976/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762674.461159:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762674.461161:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762674.461162:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762674.461164:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2649 00000100:00080000:2.0:1789762674.461167:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789762674.461173:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762674.461174:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762674.461180:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762674.461188:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512975104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762674.461225:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762674/real 1789762674] req@ffff8dc00368df80 x1876699512974720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762674.461246:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512974720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762674.461258:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762674.461262:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762674.461266:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512974720:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762674.461290:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512974848:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762674.461316:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762674.461329:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762674/real 1789762674] req@ffff8dc00368e300 x1876699512974848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762674.461343:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512974848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762674.461349:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762674.461351:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762674.461353:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512974848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762674.461358:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512974976:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762674.461371:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762674.461378:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762674/real 1789762674] req@ffff8dc00368ed80 x1876699512974976/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762674.461385:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512974976/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762674.461390:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762674.461392:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762674.461394:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512974976:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762674.461397:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512975104:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762674.461407:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762674.461413:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762674/real 1789762674] req@ffff8dc00368f800 x1876699512975104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762674.461420:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512975104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762674.461429:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762674.461431:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762674.461433:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512975104:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762675.485063:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762676.509272:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762677.533164:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762678.557064:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762679.581016:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762679.581025:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762679.581028:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762679.581037:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2654 00000100:00080000:0.0:1789762679.581047:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762679.581054:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762679.581087:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b800 x1876699512975232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762679.581110:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762679.581113:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762679.581115:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762679.581119:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2654 00000100:00080000:0.0:1789762679.581122:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762679.581125:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762679.581133:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762679.581143:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149f80 x1876699512975360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762679.581147:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512975232:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789762679.581164:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762679.581167:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762679.581169:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762679.581172:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2654 00000100:00080000:0.0:1789762679.581175:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762679.581177:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762679.581182:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762679.581191:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149180 x1876699512975488/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762679.581200:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762679.581204:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762679.581213:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762679.581218:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2654 00000100:00080000:0.0:1789762679.581224:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762679.581229:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762679.581235:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762679.581236:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512975360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762679.581260:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149880 x1876699512975616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762679.581267:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512975488:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762679.581296:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512975616:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762679.581337:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762680.605056:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762681.629080:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762682.653022:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789762682.653105:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:2.0:1789762682.653139:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789762682.653152:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789762682.653183:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789762682.653202:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789762682.653213:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1789762682.653243:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762679/real 1789762682] req@ffff8dc005149880 x1876699512975616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762682.653270:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149880 x1876699512975616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762682.653283:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762682.653289:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762682.653293:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512975616:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762682.653302:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762679/real 1789762682] req@ffff8dc005149f80 x1876699512975360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762682.653312:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149f80 x1876699512975360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762682.653318:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762682.653321:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762682.653323:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512975360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762682.653328:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762679/real 1789762682] req@ffff8dc005149180 x1876699512975488/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762682.653347:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149180 x1876699512975488/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762682.653355:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762682.653359:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762682.653362:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512975488:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762682.653369:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762679/real 1789762682] req@ffff8dc00514b800 x1876699512975232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762682.653382:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b800 x1876699512975232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762682.653396:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762682.653400:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762682.653403:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512975232:192.168.203.156@tcp:250:kworker.0 00010000:00010000:0.0:1789762683.677066:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762684.701068:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762684.701068:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762684.701078:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762684.701082:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762684.701093:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2659 00000100:00080000:3.0:1789762684.701102:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762684.701110:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762684.701142:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512975744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762684.701161:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762684.701164:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762684.701166:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762684.701169:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2659 00000100:00080000:3.0:1789762684.701171:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762684.701174:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762684.701181:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762684.701190:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512975872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762684.701198:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762684.701201:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789762684.701204:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512975744:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762684.701204:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762684.701211:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2659 00000100:00080000:3.0:1789762684.701214:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762684.701216:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762684.701220:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762684.701227:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512976000/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762684.701236:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762684.701238:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762684.701239:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762684.701242:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2659 00000100:00080000:3.0:1789762684.701244:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762684.701247:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762684.701249:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762684.701256:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512976128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762684.701285:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762684.701345:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762684/real 1789762684] req@ffff8dc0107ad180 x1876699512975744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762684.701371:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512975744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762684.701384:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762684.701388:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762684.701393:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512975744:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762684.701426:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512975872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762684.701452:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762684.701467:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762684/real 1789762684] req@ffff8dc0107af480 x1876699512975872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762684.701480:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512975872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762684.701488:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762684.701492:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762684.701494:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512975872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762684.701500:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512976000:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762684.701522:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762684.701533:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762684/real 1789762684] req@ffff8dc0107adf80 x1876699512976000/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762684.701544:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512976000/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762684.701552:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762684.701555:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762684.701558:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512976000:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762684.701566:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512976128:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762684.701589:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762684.701604:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762684/real 1789762684] req@ffff8dc0107ac380 x1876699512976128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762684.701617:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512976128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762684.701627:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762684.701631:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762684.701634:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512976128:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762685.725038:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762686.749039:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762687.773034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762688.797091:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762689.820980:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762689.820989:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762689.820992:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762689.821003:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2664 00000100:00080000:3.0:1789762689.821013:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762689.821020:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762689.821053:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512976256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762689.821073:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762689.821076:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762689.821077:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762689.821081:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2664 00000100:00080000:3.0:1789762689.821083:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762689.821087:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762689.821096:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762689.821104:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512976384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762689.821117:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512976256:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762689.821118:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762689.821131:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762689.821134:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762689.821138:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2664 00000100:00080000:3.0:1789762689.821140:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762689.821142:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762689.821145:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762689.821156:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512976512/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762689.821164:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762689.821166:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762689.821168:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762689.821171:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2664 00000100:00080000:3.0:1789762689.821173:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762689.821176:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762689.821179:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762689.821186:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512976640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762689.821210:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762689.821255:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762689/real 1789762689] req@ffff8dc0107af800 x1876699512976256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762689.821273:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512976256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762689.821286:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762689.821291:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762689.821295:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512976256:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762689.821322:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512976384:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762689.821329:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:1.0:1789762689.821343:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762689.821355:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762689/real 1789762689] req@ffff8dc0107ae680 x1876699512976384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762689.821364:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512976384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762689.821369:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762689.821372:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762689.821374:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512976384:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762689.821378:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512976512:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762689.821390:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762689.821396:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762689/real 1789762689] req@ffff8dc0107ace00 x1876699512976512/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762689.821404:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512976512/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762689.821409:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762689.821412:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762689.821414:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512976512:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762689.821417:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512976640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762689.821428:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762689.821434:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762689/real 1789762689] req@ffff8dc0107af100 x1876699512976640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762689.821441:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512976640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762689.821450:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762689.821452:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762689.821454:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512976640:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762690.845099:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762691.869052:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762692.893034:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762693.917045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762694.940999:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762694.941007:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762694.941010:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762694.941020:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2669 00000100:00080000:3.0:1789762694.941030:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762694.941035:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789762694.941058:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762694.941070:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512976768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762694.941091:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762694.941093:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762694.941095:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762694.941098:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2669 00000100:00080000:3.0:1789762694.941100:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762694.941103:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762694.941110:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762694.941121:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512976896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762694.941126:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512976768:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762694.941134:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762694.941137:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762694.941139:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762694.941142:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2669 00000100:00080000:3.0:1789762694.941144:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762694.941146:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762694.941150:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762694.941160:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512977024/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762694.941169:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762694.941171:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762694.941172:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762694.941175:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2669 00000100:00080000:3.0:1789762694.941179:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762694.941182:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762694.941184:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762694.941189:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762694.941193:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512977152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762694.941226:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762694/real 1789762694] req@ffff8dc0107aea00 x1876699512976768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762694.941249:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512976768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762694.941258:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762694.941260:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762694.941263:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512976768:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762694.941281:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512976896:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762694.941296:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762694.941304:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762694/real 1789762694] req@ffff8dc0107ad880 x1876699512976896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762694.941312:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512976896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762694.941318:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762694.941320:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762694.941322:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512976896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762694.941327:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512977024:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762694.941338:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762694.941344:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762694/real 1789762694] req@ffff8dc0107af100 x1876699512977024/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762694.941352:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512977024/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762694.941357:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762694.941359:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762694.941361:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512977024:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762694.941364:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512977152:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762694.941374:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762694.941380:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762694/real 1789762694] req@ffff8dc0107ace00 x1876699512977152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762694.941387:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512977152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762694.941392:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762694.941394:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762694.941396:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512977152:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762695.965046:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762696.989047:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762698.013044:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762699.037017:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762700.060990:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762700.060998:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762700.061002:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762700.061011:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2674 00000100:00080000:3.0:1789762700.061021:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762700.061029:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789762700.061063:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762700.061068:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512977280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762700.061095:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762700.061099:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762700.061101:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762700.061107:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2674 00000100:00080000:3.0:1789762700.061110:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762700.061113:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762700.061122:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762700.061133:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512977408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762700.061141:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762700.061144:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762700.061145:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762700.061151:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2674 00000100:00100000:1.0:1789762700.061152:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512977280:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762700.061153:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762700.061158:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762700.061165:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762700.061174:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512977536/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762700.061182:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762700.061184:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762700.061185:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762700.061188:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2674 00000100:00080000:3.0:1789762700.061190:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762700.061193:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762700.061196:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762700.061211:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512977664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762700.061229:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762700.061275:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762700/real 1789762700] req@ffff8dc0107ae680 x1876699512977280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762700.061290:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512977280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762700.061298:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762700.061301:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762700.061303:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512977280:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762700.061329:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512977408:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762700.061347:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762700.061356:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762700/real 1789762700] req@ffff8dc0107af800 x1876699512977408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762700.061364:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512977408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762700.061370:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762700.061372:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762700.061374:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512977408:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762700.061379:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512977536:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762700.061390:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762700.061396:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762700/real 1789762700] req@ffff8dc0107ac380 x1876699512977536/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762700.061404:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512977536/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762700.061414:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762700.061416:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762700.061418:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512977536:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762700.061422:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512977664:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762700.061432:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762700.061438:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762700/real 1789762700] req@ffff8dc0107adf80 x1876699512977664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762700.061445:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512977664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762700.061451:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762700.061453:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762700.061456:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512977664:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762701.085030:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762702.109013:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762703.133075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762704.157026:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762705.180984:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762705.180992:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762705.180996:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762705.181004:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2679 00000100:00080000:1.0:1789762705.181012:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762705.181018:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789762705.181032:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789762705.181048:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512977792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762705.181071:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762705.181075:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762705.181077:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762705.181080:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2679 00000100:00080000:1.0:1789762705.181083:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762705.181086:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762705.181094:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762705.181108:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512977920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762705.181114:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512977792:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789762705.181119:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762705.181123:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762705.181125:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762705.181129:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2679 00000100:00080000:1.0:1789762705.181132:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762705.181135:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762705.181138:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762705.181149:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512978048/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762705.181158:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762705.181161:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762705.181162:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762705.181166:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2679 00000100:00080000:1.0:1789762705.181170:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762705.181173:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762705.181178:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762705.181187:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762705.181188:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512978176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762705.181228:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762705/real 1789762705] req@ffff8dc005c95180 x1876699512977792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762705.181245:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512977792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762705.181257:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762705.181261:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762705.181266:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512977792:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762705.181289:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512977920:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762705.181313:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762705.181326:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762705/real 1789762705] req@ffff8dc005c97100 x1876699512977920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762705.181338:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512977920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762705.181347:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762705.181351:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762705.181355:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512977920:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762705.181362:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512978048:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762705.181385:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762705.181397:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762705/real 1789762705] req@ffff8dc005c94a80 x1876699512978048/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762705.181412:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512978048/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762705.181421:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762705.181425:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762705.181429:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512978048:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762705.181436:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512978176:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762705.181458:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762705.181475:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762705/real 1789762705] req@ffff8dc005c95c00 x1876699512978176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762705.181492:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512978176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762705.181501:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762705.181505:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762705.181511:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512978176:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762706.205107:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762707.229050:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762708.253056:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762709.277032:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762710.301006:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762710.301014:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762710.301018:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762710.301029:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2684 00000100:00080000:3.0:1789762710.301038:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762710.301044:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789762710.301061:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762710.301080:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512978304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762710.301101:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762710.301104:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762710.301106:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762710.301110:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2684 00000100:00080000:3.0:1789762710.301114:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762710.301119:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762710.301129:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762710.301145:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512978432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762710.301149:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512978304:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762710.301165:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762710.301168:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762710.301169:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762710.301174:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2684 00000100:00080000:3.0:1789762710.301177:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762710.301180:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762710.301185:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762710.301195:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512978560/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762710.301203:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762710.301206:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762710.301207:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762710.301213:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2684 00000800:00000100:0.0:1789762710.301214:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762710.301220:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762710.301223:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762710.301227:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762710.301235:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512978688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762710.301255:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762710/real 1789762710] req@ffff8dc0107af480 x1876699512978304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762710.301271:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512978304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762710.301280:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762710.301283:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762710.301286:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512978304:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762710.301305:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512978432:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762710.301318:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762710.301327:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762710/real 1789762710] req@ffff8dc0107ad180 x1876699512978432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762710.301335:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512978432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762710.301341:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762710.301343:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762710.301345:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512978432:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762710.301349:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512978560:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762710.301367:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762710.301378:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762710/real 1789762710] req@ffff8dc0107aed80 x1876699512978560/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762710.301399:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512978560/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762710.301408:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762710.301410:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762710.301413:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512978560:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762710.301417:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512978688:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762710.301433:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762710.301441:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762710/real 1789762710] req@ffff8dc0107ad500 x1876699512978688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762710.301449:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512978688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762710.301454:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762710.301456:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762710.301458:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512978688:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762711.325001:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762712.349052:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762713.373006:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762714.397034:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762715.420998:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1789762715.421007:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762715.421009:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762715.421015:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762715.421026:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2690 00000100:00080000:2.0:1789762715.421036:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762715.421043:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762715.421085:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512978816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762715.421106:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762715.421110:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762715.421112:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762715.421116:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2690 00000100:00080000:2.0:1789762715.421120:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762715.421125:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762715.421135:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762715.421144:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512978816:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762715.421147:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512978944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762715.421157:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762715.421160:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762715.421162:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762715.421165:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2690 00000100:00080000:2.0:1789762715.421168:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762715.421172:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762715.421177:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762715.421185:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512979072/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762715.421194:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762715.421197:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762715.421198:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762715.421202:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2690 00000800:00000100:0.0:1789762715.421205:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762715.421205:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762715.421209:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762715.421213:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762715.421222:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512979200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762715.421243:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762715/real 1789762715] req@ffff8dc00368ca80 x1876699512978816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762715.421261:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512978816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762715.421273:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762715.421276:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762715.421279:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512978816:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762715.421299:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512978944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762715.421317:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762715.421327:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762715/real 1789762715] req@ffff8dc00368ea00 x1876699512978944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762715.421336:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512978944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762715.421342:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762715.421344:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762715.421346:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512978944:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762715.421349:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512979072:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762715.421361:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762715.421367:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762715/real 1789762715] req@ffff8dc00368c000 x1876699512979072/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762715.421374:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512979072/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762715.421380:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762715.421382:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762715.421384:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512979072:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762715.421388:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512979200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762715.421399:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762715.421405:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762715/real 1789762715] req@ffff8dc00368ce00 x1876699512979200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762715.421412:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512979200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762715.421417:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762715.421419:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762715.421420:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512979200:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762716.445131:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762717.469061:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762718.493155:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762719.517311:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762720.541024:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762720.541059:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762720.541072:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762720.541077:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762720.541102:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2695 00000100:00080000:2.0:1789762720.541114:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762720.541121:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762720.541167:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512979328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762720.541205:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762720.541209:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762720.541211:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762720.541216:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2695 00000100:00080000:2.0:1789762720.541220:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762720.541224:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762720.541234:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762720.541258:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512979456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762720.541268:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512979328:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762720.541288:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762720.541296:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762720.541298:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762720.541308:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2695 00000100:00080000:2.0:1789762720.541314:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762720.541321:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762720.541330:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762720.541348:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512979584/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762720.541359:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789762720.541367:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762720.541369:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762720.541372:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762720.541377:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2695 00000100:00080000:2.0:1789762720.541383:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762720.541388:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762720.541395:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762720.541408:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512979712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762720.541421:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762720/real 1789762720] req@ffff8dc00368f100 x1876699512979328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762720.541443:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512979328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762720.541459:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762720.541462:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762720.541466:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512979328:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762720.541489:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512979456:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762720.541508:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762720.541517:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762720/real 1789762720] req@ffff8dc00368f480 x1876699512979456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762720.541526:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512979456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762720.541532:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762720.541534:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762720.541537:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512979456:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762720.541540:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512979584:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762720.541552:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762720.541558:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762720/real 1789762720] req@ffff8dc00368fb80 x1876699512979584/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762720.541566:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512979584/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762720.541571:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762720.541573:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762720.541575:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512979584:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762720.541580:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512979712:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762720.541591:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762720.541597:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762720/real 1789762720] req@ffff8dc00368d880 x1876699512979712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762720.541610:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512979712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762720.541615:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762720.541631:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762720.541633:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512979712:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762721.565091:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762722.589131:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762723.613074:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762724.637027:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762725.661032:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762725.661039:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762725.661043:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762725.661052:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2700 00000100:00080000:1.0:1789762725.661061:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1789762725.661066:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762725.661068:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762725.661105:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512979840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762725.661126:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762725.661129:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762725.661131:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762725.661134:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2700 00000100:00080000:1.0:1789762725.661137:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762725.661141:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762725.661149:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762725.661163:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512979840:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762725.661166:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512979968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762725.661182:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762725.661186:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762725.661189:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762725.661194:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2700 00000100:00080000:1.0:1789762725.661198:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762725.661203:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762725.661209:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762725.661221:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512980096/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762725.661230:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789762725.661230:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789762725.661236:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762725.661237:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762725.661241:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2700 00000100:00080000:1.0:1789762725.661245:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762725.661248:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762725.661252:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762725.661261:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512980224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762725.661272:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762725/real 1789762725] req@ffff8dc005c97800 x1876699512979840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762725.661292:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512979840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762725.661301:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762725.661304:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762725.661307:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512979840:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762725.661325:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512979968:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762725.661339:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762725.661348:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762725/real 1789762725] req@ffff8dc005c96a00 x1876699512979968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762725.661355:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512979968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762725.661361:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762725.661363:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762725.661365:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512979968:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762725.661368:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512980096:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762725.661379:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762725.661385:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762725/real 1789762725] req@ffff8dc005c94700 x1876699512980096/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762725.661392:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512980096/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762725.661401:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762725.661403:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762725.661405:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512980096:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762725.661408:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512980224:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762725.661420:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762725.661426:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762725/real 1789762725] req@ffff8dc005c95f80 x1876699512980224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762725.661433:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512980224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762725.661438:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762725.661440:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762725.661441:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512980224:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762726.684997:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762727.709120:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762728.733074:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762729.757013:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762730.781011:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762730.781019:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762730.781024:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762730.781035:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2705 00010000:00010000:3.0:1789762730.781038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762730.781046:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762730.781053:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762730.781088:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512980352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762730.781111:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762730.781115:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762730.781117:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762730.781120:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2705 00000100:00080000:2.0:1789762730.781122:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762730.781125:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762730.781134:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762730.781147:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512980352:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762730.781147:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512980480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762730.781161:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762730.781164:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762730.781165:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762730.781169:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2705 00000100:00080000:2.0:1789762730.781173:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762730.781177:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762730.781182:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762730.781195:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512980608/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762730.781207:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762730.781211:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1789762730.781216:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762730.781216:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762730.781225:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2705 00000100:00080000:2.0:1789762730.781230:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762730.781235:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762730.781241:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762730.781251:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512980736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762730.781262:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762730/real 1789762730] req@ffff8dc00368d880 x1876699512980352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762730.781278:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512980352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762730.781287:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762730.781290:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762730.781293:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512980352:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762730.781312:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512980480:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762730.781328:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762730.781337:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762730/real 1789762730] req@ffff8dc00368fb80 x1876699512980480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762730.781350:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512980480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762730.781356:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762730.781358:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762730.781360:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512980480:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762730.781364:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512980608:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762730.781375:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762730.781381:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762730/real 1789762730] req@ffff8dc00368f480 x1876699512980608/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762730.781388:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512980608/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762730.781393:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762730.781395:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762730.781397:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512980608:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762730.781400:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512980736:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762730.781411:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762730.781416:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762730/real 1789762730] req@ffff8dc00368f100 x1876699512980736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762755 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762730.781423:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512980736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762755 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762730.781433:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762730.781435:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762730.781437:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512980736:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762731.805149:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762732.829121:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762733.853064:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762734.877077:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762735.901057:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762735.901068:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762735.901073:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789762735.901088:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762735.901088:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2710 00000100:00080000:2.0:1789762735.901100:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762735.901110:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762735.901157:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512980864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762735.901189:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762735.901192:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762735.901193:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762735.901196:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2710 00000100:00080000:2.0:1789762735.901199:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762735.901201:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762735.901210:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762735.901220:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512980992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762735.901228:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762735.901230:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762735.901233:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762735.901236:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2710 00000100:00080000:2.0:1789762735.901238:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762735.901240:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762735.901244:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762735.901251:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512980864:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762735.901252:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512981120/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762735.901266:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762735.901268:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762735.901269:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762735.901272:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2710 00000100:00080000:2.0:1789762735.901274:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762735.901276:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762735.901279:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762735.901286:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512981248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762735.901333:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762735.901376:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762735/real 1789762735] req@ffff8dc00368ce00 x1876699512980864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762735.901392:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512980864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762735.901401:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762735.901405:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762735.901408:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512980864:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762735.901414:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512980992:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762735.901429:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762735.901437:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762735/real 1789762735] req@ffff8dc00368c000 x1876699512980992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762735.901445:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512980992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762735.901451:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762735.901454:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762735.901457:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512980992:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762735.901480:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512981120:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762735.901500:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762735.901524:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762735/real 1789762735] req@ffff8dc00368ea00 x1876699512981120/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762735.901533:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512981120/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762735.901539:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762735.901541:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762735.901544:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512981120:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762735.901548:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512981248:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762735.901567:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762735.901574:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762735/real 1789762735] req@ffff8dc00368ca80 x1876699512981248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762735.901581:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512981248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762735.901587:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762735.901589:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762735.901591:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512981248:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762736.925045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762737.949080:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762738.973011:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762739.997059:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762741.021008:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762741.021016:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762741.021019:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762741.021028:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2715 00000100:00080000:0.0:1789762741.021037:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762741.021044:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762741.021076:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149c00 x1876699512981376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762741.021095:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762741.021099:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762741.021101:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762741.021104:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2715 00000100:00080000:0.0:1789762741.021107:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762741.021110:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762741.021119:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762741.021120:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512981376:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762741.021135:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148380 x1876699512981504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762741.021144:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762741.021147:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762741.021148:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762741.021152:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2715 00000100:00080000:0.0:1789762741.021155:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762741.021158:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762741.021161:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762741.021171:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148000 x1876699512981632/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762741.021179:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789762741.021183:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1789762741.021184:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762741.021188:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762741.021192:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2715 00000100:00080000:0.0:1789762741.021196:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762741.021199:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762741.021203:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762741.021212:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b480 x1876699512981760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762741.021224:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762741/real 1789762741] req@ffff8dc005149c00 x1876699512981376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762741.021242:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149c00 x1876699512981376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762741.021252:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762741.021255:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762741.021260:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512981376:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762741.021283:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512981504:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762741.021302:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762741.021316:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762741/real 1789762741] req@ffff8dc005148380 x1876699512981504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:3.0:1789762741.021323:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762741.021329:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148380 x1876699512981504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762741.021337:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762741.021340:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762741.021343:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512981504:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762741.021347:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512981632:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762741.021361:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762741.021367:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762741/real 1789762741] req@ffff8dc005148000 x1876699512981632/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762741.021375:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148000 x1876699512981632/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762741.021381:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762741.021384:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762741.021386:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512981632:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762741.021390:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512981760:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762741.021402:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762741.021408:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762741/real 1789762741] req@ffff8dc00514b480 x1876699512981760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762741.021417:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b480 x1876699512981760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762741.021423:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762741.021426:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762741.021428:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512981760:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762742.045001:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762743.069039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762744.093063:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762745.117046:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762746.141022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762746.141024:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762746.141038:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762746.141043:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762746.141055:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2720 00000100:00080000:2.0:1789762746.141067:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762746.141074:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762746.141123:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512981888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762746.141156:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762746.141160:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762746.141163:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762746.141169:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2720 00000100:00080000:2.0:1789762746.141174:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762746.141180:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762746.141192:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762746.141195:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512981888:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762746.141213:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512982016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762746.141230:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762746.141234:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762746.141237:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762746.141242:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2720 00000100:00080000:2.0:1789762746.141247:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762746.141252:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762746.141259:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762746.141274:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512982144/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762746.141288:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762746.141293:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762746.141295:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762746.141301:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2720 00000100:00080000:2.0:1789762746.141306:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762746.141312:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762746.141318:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762746.141335:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512982272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762746.141337:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512982016:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762746.141387:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512982144:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762746.141413:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512982272:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762747.165048:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762748.189319:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762749.213122:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762749.213238:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:1.0:1789762749.213282:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762749.213297:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762749.213334:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762749.213344:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762749.213353:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:0.0:1789762749.213375:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762746/real 1789762749] req@ffff8dc00368df80 x1876699512982272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762749.213417:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512982272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762749.213428:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762749.213433:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762749.213437:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512982272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762749.213443:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762746/real 1789762749] req@ffff8dc00368ed80 x1876699512982016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762749.213454:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512982016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762749.213463:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762749.213468:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762749.213476:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512982016:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762749.213484:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762746/real 1789762749] req@ffff8dc00368e300 x1876699512982144/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762749.213502:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512982144/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762749.213511:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762749.213514:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762749.213517:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512982144:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762749.213524:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762746/real 1789762749] req@ffff8dc00368f800 x1876699512981888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762749.213537:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512981888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762749.213546:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762749.213551:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762749.213555:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512981888:192.168.203.156@tcp:250:kworker.0 00010000:00010000:2.0:1789762750.237032:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762751.260995:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762751.261004:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762751.261007:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762751.261016:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2725 00000100:00080000:0.0:1789762751.261025:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762751.261032:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762751.261069:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514bb80 x1876699512982400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762751.261119:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762751.261124:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762751.261126:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762751.261132:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2725 00000100:00080000:0.0:1789762751.261136:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762751.261140:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762751.261157:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762751.261158:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512982400:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762751.261174:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514ad80 x1876699512982528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762751.261201:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762751.261204:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762751.261206:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762751.261209:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2725 00000100:00080000:0.0:1789762751.261213:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762751.261218:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762751.261223:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:2.0:1789762751.261237:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762751.261239:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148700 x1876699512982656/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762751.261252:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762751.261257:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762751.261259:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762751.261264:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2725 00000100:00080000:0.0:1789762751.261268:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762751.261274:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762751.261279:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762751.261281:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762751/real 1789762751] req@ffff8dc00514bb80 x1876699512982400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762751.261294:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514aa00 x1876699512982784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762751.261305:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514bb80 x1876699512982400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762751.261320:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762751.261326:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762751.261331:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512982400:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762751.261365:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512982528:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762751.261404:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762751.261418:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762751/real 1789762751] req@ffff8dc00514ad80 x1876699512982528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762751.261431:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514ad80 x1876699512982528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762751.261439:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762751.261443:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762751.261446:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512982528:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762751.261452:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512982656:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762751.261473:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762751.261486:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762751/real 1789762751] req@ffff8dc005148700 x1876699512982656/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762751.261501:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148700 x1876699512982656/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1789762751.261510:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762751.261511:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762751.261518:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762751.261523:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512982656:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762751.261530:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512982784:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762751.261553:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762751.261565:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762751/real 1789762751] req@ffff8dc00514aa00 x1876699512982784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762751.261578:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514aa00 x1876699512982784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762751.261586:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762751.261589:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762751.261592:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512982784:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762752.285022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762753.309033:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762754.333051:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762755.357090:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762756.381034:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762756.381035:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762756.381043:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762756.381047:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762756.381056:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2730 00000100:00080000:1.0:1789762756.381067:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762756.381073:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762756.381105:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512982912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762756.381127:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762756.381131:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762756.381134:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762756.381138:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2730 00000100:00080000:1.0:1789762756.381142:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762756.381146:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762756.381155:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762756.381158:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512982912:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762756.381169:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512983040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762756.381179:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762756.381182:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762756.381184:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762756.381187:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2730 00000100:00080000:1.0:1789762756.381191:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762756.381194:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762756.381200:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762756.381208:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512983168/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762756.381215:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789762756.381218:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762756.381221:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762756.381223:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762756.381226:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2730 00000100:00080000:1.0:1789762756.381229:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762756.381233:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762756.381237:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762756.381245:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512983296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762756.381253:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762756/real 1789762756] req@ffff8dc005c94380 x1876699512982912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762756.381279:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512982912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762756.381290:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762756.381294:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762756.381297:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512982912:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762756.381317:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512983040:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762756.381335:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762756.381344:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762756/real 1789762756] req@ffff8dc005c97b80 x1876699512983040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762756.381376:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512983040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762756.381382:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762756.381384:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762756.381387:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512983040:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762756.381391:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512983168:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762756.381405:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762756.381412:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762756/real 1789762756] req@ffff8dc005c96300 x1876699512983168/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762756.381420:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512983168/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762756.381430:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762756.381432:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762756.381434:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512983168:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762756.381437:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512983296:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762756.381447:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762756.381453:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762756/real 1789762756] req@ffff8dc005c95500 x1876699512983296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762756.381460:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512983296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762756.381466:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762756.381467:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762756.381469:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512983296:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762757.405061:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762758.429061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762759.453088:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762760.477156:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762761.501062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762761.501084:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762761.501094:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762761.501100:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762761.501113:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2736 00000100:00080000:1.0:1789762761.501126:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762761.501135:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762761.501181:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512983424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762761.501210:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762761.501214:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762761.501217:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762761.501223:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2736 00000100:00080000:1.0:1789762761.501227:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762761.501232:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762761.501244:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762761.501257:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512983424:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762761.501262:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512983552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762761.501281:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762761.501284:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762761.501286:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762761.501290:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2736 00000100:00080000:1.0:1789762761.501294:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762761.501298:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762761.501304:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762761.501339:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512983680/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762761.501351:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762761.501355:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762761.501357:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789762761.501364:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789762761.501376:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2736 00000100:00080000:1.0:1789762761.501384:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762761.501389:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762761.501396:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762761.501412:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512983808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762761.501428:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762761/real 1789762761] req@ffff8dc005c94e00 x1876699512983424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762761.501463:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512983424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762761.501495:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762761.501504:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762761.501511:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512983424:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762761.501543:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512983552:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762761.501574:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762761.501590:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762761/real 1789762761] req@ffff8dc005c97480 x1876699512983552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762761.501604:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512983552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762761.501614:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762761.501619:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762761.501624:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512983552:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762761.501635:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512983680:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762761.501669:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762761.501689:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762761/real 1789762761] req@ffff8dc005c96680 x1876699512983680/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762761.501733:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512983680/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762761.501744:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762761.501748:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762761.501751:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512983680:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762761.501759:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512983808:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762761.501785:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762761.501798:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762761/real 1789762761] req@ffff8dc005c95880 x1876699512983808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762761.501811:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512983808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762761.501820:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762761.501824:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762761.501827:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512983808:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762762.525124:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762763.549124:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762764.573095:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762765.597072:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762766.621094:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762766.621108:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762766.621118:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762766.621125:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762766.621139:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2741 00000100:00080000:2.0:1789762766.621152:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762766.621162:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762766.621209:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512983936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762766.621248:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762766.621253:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762766.621255:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762766.621261:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2741 00000100:00080000:2.0:1789762766.621266:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762766.621292:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762766.621306:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762766.621321:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512983936:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762766.621326:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512984064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762766.621345:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762766.621349:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762766.621352:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762766.621356:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2741 00000100:00080000:2.0:1789762766.621360:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762766.621364:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762766.621371:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762766.621385:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512984192/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762766.621397:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762766.621402:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762766.621408:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789762766.621409:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762766.621418:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2741 00000100:00080000:2.0:1789762766.621427:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762766.621433:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762766.621439:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762766.621453:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512984320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762766.621468:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762766/real 1789762766] req@ffff8dc00368d180 x1876699512983936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762766.621494:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512983936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762766.621508:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762766.621514:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762766.621519:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512983936:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762766.621552:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512984064:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762766.621578:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762766.621592:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762766/real 1789762766] req@ffff8dc00368c700 x1876699512984064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762766.621608:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512984064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762766.621618:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762766.621622:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762766.621626:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512984064:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762766.621633:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512984192:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762766.621653:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762766.621664:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762766/real 1789762766] req@ffff8dc00368dc00 x1876699512984192/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762766.621677:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512984192/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762766.621687:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762766.621691:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762766.621694:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512984192:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762766.621701:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512984320:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762766.621719:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762766.621730:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762766/real 1789762766] req@ffff8dc00368e680 x1876699512984320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762766.621760:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512984320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762766.621770:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762766.621773:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762766.621777:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512984320:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762767.645153:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762768.669096:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762769.693111:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762770.717009:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762771.741036:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762771.741047:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762771.741052:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762771.741064:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2746 00000100:00080000:3.0:1789762771.741076:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762771.741086:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789762771.741114:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762771.741131:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512984448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762771.741168:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762771.741172:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762771.741175:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762771.741179:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2746 00000100:00080000:3.0:1789762771.741183:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762771.741187:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762771.741197:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762771.741210:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512984576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762771.741223:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762771.741227:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762771.741230:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762771.741256:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2746 00000100:00080000:3.0:1789762771.741261:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762771.741265:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762771.741271:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762771.741274:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512984448:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762771.741288:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512984704/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762771.741303:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762771.741307:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762771.741309:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762771.741314:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2746 00000100:00080000:3.0:1789762771.741318:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762771.741322:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762771.741327:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762771.741340:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512984832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762771.741355:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762771.741402:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762771/real 1789762771] req@ffff8dc0107ac000 x1876699512984448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762771.741423:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512984448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762771.741436:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762771.741440:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762771.741445:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512984448:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762771.741453:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512984576:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762771.741474:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762771.741486:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762771/real 1789762771] req@ffff8dc0107ac700 x1876699512984576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762771.741499:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512984576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762771.741516:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762771.741519:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762771.741523:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512984576:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762771.741546:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512984704:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762771.741564:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762771.741575:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762771/real 1789762771] req@ffff8dc0107ae300 x1876699512984704/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762771.741587:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512984704/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762771.741595:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762771.741599:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762771.741602:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512984704:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762771.741608:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512984832:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762771.741624:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762771.741633:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762771/real 1789762771] req@ffff8dc0107aca80 x1876699512984832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762796 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762771.741645:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512984832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762796 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762771.741653:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762771.741655:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762771.741660:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512984832:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762772.765053:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762773.789074:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762774.813032:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762775.837056:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762776.861163:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762776.861220:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762776.861228:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762776.861233:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762776.861245:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2751 00000100:00080000:0.0:1789762776.861268:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762776.861276:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762776.861312:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148a80 x1876699512984960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762776.861340:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762776.861344:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762776.861347:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762776.861353:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2751 00000100:00080000:0.0:1789762776.861357:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762776.861371:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762776.861371:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512984960:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1789762776.861388:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762776.861406:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514aa00 x1876699512985088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762776.861418:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762776.861423:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762776.861425:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762776.861430:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2751 00000100:00080000:0.0:1789762776.861435:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762776.861441:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1789762776.861447:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:0.0:1789762776.861451:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762776.861466:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148700 x1876699512985216/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762776.861478:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762776.861483:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1789762776.861486:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762776/real 1789762776] req@ffff8dc005148a80 x1876699512984960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762776.861487:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762776.861497:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2751 00000100:00080000:0.0:1789762776.861504:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762776.861509:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148a80 x1876699512984960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762776.861512:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762776.861522:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762776.861525:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762776.861530:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762776.861534:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512984960:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762776.861539:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514ad80 x1876699512985344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762776.861558:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512985088:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762776.861589:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762776.861598:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762776/real 1789762776] req@ffff8dc00514aa00 x1876699512985088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762776.861607:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514aa00 x1876699512985088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762776.861614:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762776.861616:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762776.861620:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512985088:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762776.861624:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512985216:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762776.861647:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762776.861654:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762776/real 1789762776] req@ffff8dc005148700 x1876699512985216/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762776.861662:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148700 x1876699512985216/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762776.861668:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762776.861671:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762776.861675:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512985216:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762776.861679:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512985344:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762776.861691:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762776.861700:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762776/real 1789762776] req@ffff8dc00514ad80 x1876699512985344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762776.861708:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514ad80 x1876699512985344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762776.861714:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762776.861717:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762776.861719:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512985344:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762777.885075:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762778.909055:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762779.933018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762780.957032:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762781.981009:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762781.981021:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762781.981027:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762781.981041:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2756 00000100:00080000:2.0:1789762781.981054:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762781.981063:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789762781.981089:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762781.981113:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699512985472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762781.981146:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762781.981150:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762781.981152:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762781.981171:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2756 00000100:00080000:2.0:1789762781.981176:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762781.981181:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762781.981192:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762781.981206:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512985472:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762781.981210:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699512985600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762781.981231:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762781.981236:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762781.981238:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762781.981244:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2756 00000100:00080000:2.0:1789762781.981248:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762781.981252:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762781.981259:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762781.981272:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699512985728/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762781.981281:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762781.981297:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762781.981302:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762781.981304:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762781.981309:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2756 00000100:00080000:2.0:1789762781.981315:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762781.981321:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762781.981326:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762781.981328:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762781/real 1789762781] req@ffff8dc00368e680 x1876699512985472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762781.981344:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699512985856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762781.981350:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699512985472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762781.981413:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762781.981419:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762781.981423:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512985472:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762781.981450:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512985600:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762781.981467:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762781.981476:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762781/real 1789762781] req@ffff8dc00368dc00 x1876699512985600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762781.981485:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699512985600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762781.981491:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762781.981493:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762781.981495:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512985600:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762781.981499:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512985728:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762781.981511:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762781.981517:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762781/real 1789762781] req@ffff8dc00368c700 x1876699512985728/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762781.981524:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699512985728/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762781.981530:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762781.981532:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762781.981534:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512985728:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762781.981538:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512985856:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762781.981549:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762781.981555:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762781/real 1789762781] req@ffff8dc00368d180 x1876699512985856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762781.981562:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699512985856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762781.981568:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762781.981571:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762781.981574:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512985856:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762783.005000:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762784.029376:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762785.053035:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762786.077101:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762787.101080:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1789762787.101081:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762787.101094:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762787.101099:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762787.101112:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2761 00000100:00080000:1.0:1789762787.101140:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762787.101147:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762787.101205:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96d80 x1876699512985984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762787.101234:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762787.101237:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762787.101257:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762787.101266:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2761 00000100:00080000:1.0:1789762787.101271:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762787.101277:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762787.101290:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762787.101322:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512985984:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762787.101323:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512986112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762787.101353:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762787.101356:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762787.101358:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762787.101361:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2761 00000100:00080000:1.0:1789762787.101366:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762787.101370:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762787.101374:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762787.101386:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512986240/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762787.101394:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762787.101397:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762787.101398:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762787.101401:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2761 00000100:00080000:1.0:1789762787.101404:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762787.101407:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762787.101410:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762787.101421:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512986368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762787.101424:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762787.101512:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762787/real 1789762787] req@ffff8dc005c96d80 x1876699512985984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762787.101552:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96d80 x1876699512985984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762787.101564:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762787.101569:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762787.101575:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512985984:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762787.101617:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512986112:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762787.101645:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762787.101675:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762787/real 1789762787] req@ffff8dc005c95880 x1876699512986112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762787.101689:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512986112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762787.101705:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762787.101709:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762787.101713:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512986112:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762787.101720:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512986240:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762787.101742:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762787.101755:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762787/real 1789762787] req@ffff8dc005c96680 x1876699512986240/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762787.101769:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512986240/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762787.101779:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762787.101783:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762787.101787:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512986240:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762787.101793:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512986368:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762787.101813:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762787.101822:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762787/real 1789762787] req@ffff8dc005c97480 x1876699512986368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762787.101830:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512986368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762787.101835:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762787.101837:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762787.101839:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512986368:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762788.125055:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762789.149022:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762790.173060:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762791.197150:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762792.220983:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762792.220992:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762792.220995:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762792.221005:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2766 00000100:00080000:1.0:1789762792.221013:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762792.221026:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789762792.221046:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789762792.221061:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512986496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762792.221093:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762792.221098:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762792.221100:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762792.221104:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2766 00000100:00080000:1.0:1789762792.221107:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762792.221111:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762792.221119:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762792.221124:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512986496:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762792.221134:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512986624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762792.221142:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762792.221144:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762792.221145:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762792.221149:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2766 00000100:00080000:1.0:1789762792.221151:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762792.221153:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762792.221157:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762792.221165:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512986752/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762792.221171:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762792.221174:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762792.221175:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762792.221178:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2766 00000100:00080000:1.0:1789762792.221188:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762792.221191:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762792.221195:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762792.221204:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512986880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762792.221206:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762792.221255:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762792/real 1789762792] req@ffff8dc005c94e00 x1876699512986496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762792.221270:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512986496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762792.221278:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762792.221281:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762792.221284:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512986496:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762792.221302:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512986624:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762792.221318:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762792.221327:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762792/real 1789762792] req@ffff8dc005c95500 x1876699512986624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762792.221335:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512986624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762792.221340:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762792.221342:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762792.221345:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512986624:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762792.221348:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512986752:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762792.221359:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762792.221366:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762792/real 1789762792] req@ffff8dc005c96300 x1876699512986752/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762792.221373:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512986752/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762792.221378:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762792.221380:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762792.221382:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512986752:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762792.221385:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512986880:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762792.221396:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762792.221402:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762792/real 1789762792] req@ffff8dc005c97b80 x1876699512986880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762792.221409:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512986880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762792.221414:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762792.221416:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762792.221417:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512986880:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762793.245067:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762794.269034:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762795.293154:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762796.317091:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762797.340968:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762797.340978:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762797.340982:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762797.340992:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2771 00000100:00080000:1.0:1789762797.341000:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762797.341007:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762797.341060:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512987008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1789762797.341085:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762797.341097:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762797.341101:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762797.341103:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762797.341109:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2771 00000100:00080000:1.0:1789762797.341113:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762797.341116:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762797.341125:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762797.341138:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512987008:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762797.341142:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512987136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762797.341153:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762797.341156:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762797.341157:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762797.341159:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2771 00000100:00080000:1.0:1789762797.341162:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762797.341164:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762797.341168:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762797.341177:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512987264/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762797.341184:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762797.341187:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762797.341189:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762797.341191:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2771 00000100:00080000:1.0:1789762797.341193:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762797.341196:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762797.341199:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762797.341206:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512987392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762797.341212:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762797.341251:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762797/real 1789762797] req@ffff8dc005c94380 x1876699512987008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762797.341264:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512987008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762797.341272:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762797.341275:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762797.341278:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512987008:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762797.341296:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512987136:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762797.341310:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762797.341321:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762797/real 1789762797] req@ffff8dc005c95f80 x1876699512987136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762797.341334:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512987136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762797.341343:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762797.341346:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762797.341350:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512987136:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762797.341356:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512987264:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762797.341378:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762797.341391:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762797/real 1789762797] req@ffff8dc005c94700 x1876699512987264/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762797.341400:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512987264/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762797.341406:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762797.341408:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762797.341410:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512987264:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762797.341414:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512987392:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762797.341426:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762797.341433:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762797/real 1789762797] req@ffff8dc005c96a00 x1876699512987392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762797.341440:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512987392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762797.341445:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762797.341447:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762797.341450:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512987392:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762798.365078:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762799.389087:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762800.413096:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762801.437076:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762802.461037:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762802.461052:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762802.461058:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762802.461074:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2777 00000100:00080000:1.0:1789762802.461087:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762802.461095:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789762802.461107:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789762802.461145:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512987520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762802.461171:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762802.461177:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762802.461178:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762802.461181:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2777 00000100:00080000:1.0:1789762802.461185:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762802.461188:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762802.461197:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762802.461215:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512987648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762802.461216:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512987520:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789762802.461230:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762802.461233:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762802.461234:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762802.461237:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2777 00000100:00080000:1.0:1789762802.461239:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762802.461241:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762802.461245:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762802.461254:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512987776/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762802.461261:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762802.461264:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762802.461266:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762802.461268:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2777 00000100:00080000:1.0:1789762802.461271:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762802.461274:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762802.461278:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762802.461284:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512987904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762802.461305:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762802.461356:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762802/real 1789762802] req@ffff8dc005c97800 x1876699512987520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762802.461378:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512987520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762802.461392:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762802.461397:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762802.461402:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512987520:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762802.461430:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512987648:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762802.461456:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762802.461484:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762802/real 1789762802] req@ffff8dc005c95c00 x1876699512987648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762802.461499:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512987648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762802.461513:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762802.461517:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762802.461520:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512987648:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762802.461527:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512987776:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762802.461550:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762802.461563:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762802/real 1789762802] req@ffff8dc005c94a80 x1876699512987776/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762802.461576:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512987776/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762802.461584:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762802.461587:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762802.461591:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512987776:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762802.461597:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512987904:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762802.461617:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762802.461628:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762802/real 1789762802] req@ffff8dc005c97100 x1876699512987904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762802.461640:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512987904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762802.461649:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762802.461653:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762802.461657:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512987904:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762803.485082:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762804.509058:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762805.533049:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762806.557071:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762807.581031:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762807.581040:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762807.581044:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1789762807.581051:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762807.581053:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2782 00000100:00080000:3.0:1789762807.581063:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762807.581069:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762807.581102:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699512988032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762807.581123:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762807.581126:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762807.581127:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762807.581130:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2782 00000100:00080000:3.0:1789762807.581133:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762807.581136:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762807.581144:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762807.581156:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512988032:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762807.581156:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699512988160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762807.581171:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762807.581173:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762807.581174:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762807.581177:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2782 00000100:00080000:3.0:1789762807.581179:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762807.581181:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762807.581185:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762807.581196:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699512988288/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762807.581202:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762807.581205:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762807.581206:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762807.581209:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2782 00000100:00080000:3.0:1789762807.581211:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762807.581213:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762807.581217:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789762807.581219:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762807.581227:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699512988416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762807.581263:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762807/real 1789762807] req@ffff8dc0107afb80 x1876699512988032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762807.581277:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699512988032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762807.581285:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762807.581288:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762807.581291:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512988032:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762807.581309:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512988160:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762807.581324:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762807.581332:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762807/real 1789762807] req@ffff8dc0107adc00 x1876699512988160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762807.581340:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699512988160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762807.581346:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762807.581348:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762807.581351:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512988160:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762807.581354:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512988288:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762807.581367:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762807.581373:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762807/real 1789762807] req@ffff8dc0107aca80 x1876699512988288/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762807.581381:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699512988288/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762807.581387:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762807.581389:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762807.581391:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512988288:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762807.581394:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512988416:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762807.581404:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762807.581409:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762807/real 1789762807] req@ffff8dc0107ae300 x1876699512988416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762807.581416:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699512988416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762807.581421:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762807.581423:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762807.581425:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512988416:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762808.605038:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762809.629112:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762810.653087:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762811.677039:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762812.700965:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762812.700973:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762812.700978:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762812.700987:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2787 00000100:00080000:3.0:1789762812.700997:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762812.701003:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762812.701035:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699512988544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762812.701053:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762812.701056:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762812.701058:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762812.701061:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2787 00010000:00010000:2.0:1789762812.701064:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762812.701064:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762812.701071:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762812.701073:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512988544:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789762812.701080:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762812.701092:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699512988672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762812.701100:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762812.701102:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762812.701104:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762812.701106:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2787 00000100:00080000:3.0:1789762812.701110:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762812.701112:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762812.701116:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762812.701124:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699512988800/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762812.701136:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762812.701139:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762812.701140:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762812.701143:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2787 00000100:00080000:3.0:1789762812.701145:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762812.701148:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762812.701151:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762812.701158:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699512988928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762812.701175:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512988672:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762812.701208:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512988800:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762812.701228:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512988928:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762813.725056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762814.749051:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762815.773051:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789762815.773181:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:1.0:1789762815.773223:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789762815.773240:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762815.773275:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762815.773288:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789762815.773299:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:0.0:1789762815.773310:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762812/real 1789762815] req@ffff8dc0107ac000 x1876699512988672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762815.773338:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699512988672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762815.773353:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762815.773359:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762815.773364:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512988672:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762815.773372:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762812/real 1789762815] req@ffff8dc0107ad500 x1876699512988800/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762815.773386:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699512988800/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762815.773395:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762815.773398:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762815.773402:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512988800:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762815.773410:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762812/real 1789762815] req@ffff8dc0107aed80 x1876699512988928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762815.773422:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699512988928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762815.773430:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762815.773435:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762815.773438:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512988928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762815.773446:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762812/real 1789762815] req@ffff8dc0107ac700 x1876699512988544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762815.773459:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699512988544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762815.773467:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762815.773470:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762815.773474:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512988544:192.168.203.156@tcp:250:kworker.0 00010000:00010000:3.0:1789762816.797091:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762817.821037:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762817.821075:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762817.821084:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762817.821089:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762817.821100:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2792 00000100:00080000:1.0:1789762817.821112:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762817.821119:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762817.821157:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699512989056/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762817.821187:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762817.821191:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762817.821193:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762817.821197:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2792 00000100:00080000:1.0:1789762817.821202:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762817.821206:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762817.821217:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762817.821238:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699512989184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762817.821244:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512989056:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789762817.821259:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762817.821263:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762817.821265:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762817.821270:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2792 00000100:00080000:1.0:1789762817.821273:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762817.821277:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762817.821283:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762817.821296:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699512989312/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762817.821306:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762817.821310:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762817.821315:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789762817.821316:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789762817.821325:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2792 00000100:00080000:1.0:1789762817.821331:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762817.821336:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762817.821342:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762817.821355:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699512989440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762817.821393:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762817/real 1789762817] req@ffff8dc005c95180 x1876699512989056/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762817.821412:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699512989056/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762817.821425:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762817.821429:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762817.821434:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512989056:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762817.821462:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512989184:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762817.821486:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762817.821500:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762817/real 1789762817] req@ffff8dc005c97100 x1876699512989184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762817.821513:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699512989184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762817.821523:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762817.821527:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762817.821531:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512989184:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762817.821538:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512989312:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762817.821559:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762817.821570:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762817/real 1789762817] req@ffff8dc005c94a80 x1876699512989312/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762817.821583:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699512989312/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762817.821593:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762817.821597:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762817.821601:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512989312:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762817.821608:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512989440:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762817.821627:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762817.821643:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762817/real 1789762817] req@ffff8dc005c95c00 x1876699512989440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762817.821656:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699512989440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762817.821665:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762817.821669:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762817.821674:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512989440:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762818.845035:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762819.869072:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762820.892992:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762821.917015:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762822.941022:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762822.941033:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762822.941037:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1789762822.941046:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762822.941050:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2797 00000100:00080000:1.0:1789762822.941063:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762822.941070:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762822.941112:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699512989568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762822.941139:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762822.941142:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762822.941143:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762822.941147:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2797 00000100:00080000:1.0:1789762822.941149:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762822.941152:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762822.941162:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762822.941176:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512989696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762822.941181:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512989568:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789762822.941210:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762822.941213:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762822.941214:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762822.941219:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2797 00000100:00080000:1.0:1789762822.941221:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762822.941224:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762822.941227:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762822.941236:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512989824/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762822.941246:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762822.941250:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762822.941253:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762822.941259:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2797 00000100:00080000:1.0:1789762822.941265:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762822.941269:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789762822.941276:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762822.941284:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762822.941303:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512989952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762822.941322:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762822/real 1789762822] req@ffff8dc005c97800 x1876699512989568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762822.941345:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699512989568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762822.941361:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762822.941365:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762822.941368:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512989568:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762822.941389:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512989696:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762822.941404:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762822.941413:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762822/real 1789762822] req@ffff8dc005c96a00 x1876699512989696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762822.941421:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512989696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762822.941427:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762822.941429:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762822.941431:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512989696:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762822.941436:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512989824:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762822.941450:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762822.941456:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762822/real 1789762822] req@ffff8dc005c94700 x1876699512989824/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762822.941464:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512989824/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762822.941470:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762822.941472:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762822.941475:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512989824:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762822.941478:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512989952:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762822.941489:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762822.941494:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762822/real 1789762822] req@ffff8dc005c95f80 x1876699512989952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762822.941502:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512989952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762822.941507:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762822.941509:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762822.941511:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512989952:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762823.965105:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762824.989031:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762826.012994:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762827.037174:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762828.060980:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762828.060991:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762828.060996:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762828.061008:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2802 00000100:00080000:1.0:1789762828.061019:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1789762828.061019:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762828.061027:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762828.061060:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512990080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762828.061081:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762828.061085:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762828.061087:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762828.061091:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2802 00000100:00080000:1.0:1789762828.061095:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762828.061100:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762828.061116:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512990080:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1789762828.061121:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762828.061134:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512990208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762828.061144:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762828.061148:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762828.061149:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762828.061153:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2802 00000100:00080000:1.0:1789762828.061165:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762828.061168:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762828.061173:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789762828.061177:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762828.061184:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512990336/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762828.061194:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762828.061197:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762828.061199:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762828.061202:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2802 00000100:00080000:1.0:1789762828.061205:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762828.061209:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762828.061213:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762828.061220:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762828/real 1789762828] req@ffff8dc005c94380 x1876699512990080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762828.061222:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512990464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762828.061242:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512990080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762828.061254:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762828.061258:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762828.061262:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512990080:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762828.061286:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512990208:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762828.061305:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762828.061323:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762828/real 1789762828] req@ffff8dc005c97b80 x1876699512990208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762828.061331:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512990208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762828.061337:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762828.061339:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762828.061342:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512990208:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762828.061346:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512990336:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762828.061358:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762828.061365:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762828/real 1789762828] req@ffff8dc005c96300 x1876699512990336/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762828.061373:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512990336/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762828.061382:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762828.061385:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762828.061387:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512990336:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762828.061390:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512990464:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762828.061400:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762828.061405:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762828/real 1789762828] req@ffff8dc005c95500 x1876699512990464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762828.061418:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512990464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762828.061423:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762828.061425:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762828.061427:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512990464:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762829.085058:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762830.109032:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762831.133077:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762832.157015:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762833.181046:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762833.181057:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762833.181062:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762833.181073:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2807 00000100:00080000:2.0:1789762833.181087:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762833.181095:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789762833.181103:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762833.181146:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699512990592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762833.181180:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762833.181185:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762833.181187:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762833.181193:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2807 00000100:00080000:2.0:1789762833.181197:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762833.181201:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762833.181211:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762833.181222:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512990592:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762833.181226:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699512990720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762833.181245:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762833.181248:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762833.181251:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762833.181257:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2807 00000100:00080000:2.0:1789762833.181259:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762833.181262:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762833.181267:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762833.181277:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699512990848/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762833.181284:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762833.181288:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762833.181290:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762833.181293:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2807 00000100:00080000:2.0:1789762833.181296:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762833.181301:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789762833.181303:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:2.0:1789762833.181312:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762833.181326:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699512990976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762833.181353:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762833/real 1789762833] req@ffff8dc00368f800 x1876699512990592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762833.181368:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699512990592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762833.181377:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762833.181381:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762833.181384:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512990592:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762833.181404:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512990720:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762833.181419:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762833.181427:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762833/real 1789762833] req@ffff8dc00368e300 x1876699512990720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762833.181435:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699512990720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762833.181440:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762833.181442:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762833.181445:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512990720:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762833.181449:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512990848:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762833.181460:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762833.181466:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762833/real 1789762833] req@ffff8dc00368ed80 x1876699512990848/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762833.181473:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699512990848/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762833.181478:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762833.181480:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762833.181482:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512990848:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762833.181486:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512990976:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762833.181497:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762833.181503:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762833/real 1789762833] req@ffff8dc00368df80 x1876699512990976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762833.181511:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699512990976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762833.181515:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762833.181517:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762833.181519:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512990976:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762834.205135:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762835.229046:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762836.253024:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762837.277021:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762838.301011:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762838.301024:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762838.301028:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762838.301038:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2812 00000100:00080000:3.0:1789762838.301048:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762838.301054:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789762838.301058:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762838.301089:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699512991104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762838.301114:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762838.301116:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762838.301118:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762838.301120:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2812 00000100:00080000:3.0:1789762838.301123:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762838.301126:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762838.301134:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762838.301143:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699512991232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762838.301147:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512991104:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762838.301159:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762838.301162:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762838.301163:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762838.301166:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2812 00000100:00080000:3.0:1789762838.301168:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762838.301170:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762838.301174:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762838.301184:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699512991360/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762838.301191:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762838.301193:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762838.301194:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762838.301197:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2812 00000100:00080000:3.0:1789762838.301201:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762838.301204:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762838.301207:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762838.301210:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762838.301220:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699512991488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762838.301256:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762838/real 1789762838] req@ffff8dc0107ad180 x1876699512991104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762838.301276:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699512991104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762838.301289:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762838.301293:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762838.301296:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512991104:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762838.301316:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512991232:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762838.301334:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762838.301343:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762838/real 1789762838] req@ffff8dc0107af480 x1876699512991232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762838.301356:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699512991232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762838.301362:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762838.301364:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762838.301366:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512991232:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762838.301370:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512991360:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762838.301382:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762838.301388:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762838/real 1789762838] req@ffff8dc0107adf80 x1876699512991360/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762838.301396:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699512991360/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762838.301401:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762838.301403:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762838.301405:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512991360:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762838.301408:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512991488:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762838.301418:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762838.301423:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762838/real 1789762838] req@ffff8dc0107ac380 x1876699512991488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762838.301430:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699512991488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762838.301440:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762838.301442:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762838.301444:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512991488:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762839.325042:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762840.349016:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762841.373057:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762842.397064:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762843.421028:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762843.421040:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762843.421044:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1789762843.421049:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762843.421055:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2818 00000100:00080000:0.0:1789762843.421067:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762843.421073:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762843.421117:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514bb80 x1876699512991616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762843.421141:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762843.421144:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762843.421146:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762843.421149:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2818 00000100:00080000:0.0:1789762843.421152:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762843.421154:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762843.421163:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762843.421172:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b480 x1876699512991744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762843.421179:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762843.421183:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762843.421184:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762843.421188:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2818 00000100:00080000:0.0:1789762843.421190:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762843.421195:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762843.421197:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512991616:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1789762843.421199:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762843.421217:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148000 x1876699512991872/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762843.421230:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762843.421234:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762843.421237:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762843.421242:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2818 00000100:00080000:0.0:1789762843.421247:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762843.421251:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762843.421257:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762843.421272:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148380 x1876699512992000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762843.421279:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762843.421336:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762843/real 1789762843] req@ffff8dc00514bb80 x1876699512991616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762843.421358:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514bb80 x1876699512991616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762843.421367:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762843.421370:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762843.421373:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512991616:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762843.421379:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512991744:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762843.421398:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762843.421408:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762843/real 1789762843] req@ffff8dc00514b480 x1876699512991744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762843.421417:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b480 x1876699512991744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762843.421422:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762843.421425:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762843.421427:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512991744:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762843.421449:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512991872:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762843.421468:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762843.421482:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762843/real 1789762843] req@ffff8dc005148000 x1876699512991872/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762843.421494:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148000 x1876699512991872/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762843.421503:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762843.421506:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762843.421509:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512991872:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762843.421515:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512992000:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762843.421534:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762843.421545:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762843/real 1789762843] req@ffff8dc005148380 x1876699512992000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762843.421556:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148380 x1876699512992000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762843.421565:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762843.421568:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762843.421571:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512992000:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762844.445010:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762845.469012:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762846.493057:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762847.517062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762848.540995:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762848.541002:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762848.541006:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762848.541015:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2823 00000100:00080000:0.0:1789762848.541024:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762848.541030:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762848.541069:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149c00 x1876699512992128/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762848.541090:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762848.541092:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762848.541094:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762848.541097:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2823 00000100:00080000:0.0:1789762848.541099:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762848.541103:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762848.541113:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762848.541125:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512992128:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762848.541130:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b800 x1876699512992256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762848.541144:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762848.541148:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762848.541150:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762848.541154:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2823 00000100:00080000:0.0:1789762848.541158:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762848.541163:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762848.541168:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762848.541179:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149180 x1876699512992384/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762848.541187:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762848.541191:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1789762848.541194:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1789762848.541195:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762848.541201:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2823 00000100:00080000:0.0:1789762848.541206:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762848.541209:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762848.541213:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762848.541222:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149f80 x1876699512992512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762848.541242:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762848/real 1789762848] req@ffff8dc005149c00 x1876699512992128/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762848.541259:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149c00 x1876699512992128/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762848.541274:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762848.541278:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762848.541283:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512992128:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762848.541313:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512992256:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762848.541339:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762848.541354:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762848/real 1789762848] req@ffff8dc00514b800 x1876699512992256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1789762848.541366:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762848.541367:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b800 x1876699512992256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762848.541378:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762848.541381:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762848.541384:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512992256:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762848.541392:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512992384:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762848.541412:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762848.541422:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762848/real 1789762848] req@ffff8dc005149180 x1876699512992384/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762848.541431:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149180 x1876699512992384/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762848.541438:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762848.541441:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762848.541445:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512992384:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762848.541449:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512992512:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762848.541464:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762848.541472:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762848/real 1789762848] req@ffff8dc005149f80 x1876699512992512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762848.541480:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149f80 x1876699512992512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762848.541486:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762848.541489:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762848.541491:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512992512:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762849.565020:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762850.589082:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762851.613082:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762852.637076:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762853.660991:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762853.661001:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762853.661006:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762853.661017:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2828 00000100:00080000:0.0:1789762853.661027:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762853.661035:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762853.661070:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149880 x1876699512992640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762853.661093:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762853.661096:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762853.661098:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762853.661103:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2828 00000100:00080000:0.0:1789762853.661108:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762853.661113:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762853.661123:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512992640:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1789762853.661124:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762853.661145:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514a300 x1876699512992768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762853.661154:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762853.661157:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762853.661159:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762853.661162:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2828 00000100:00080000:0.0:1789762853.661165:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762853.661168:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762853.661171:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762853.661180:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148e00 x1876699512992896/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762853.661185:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1789762853.661193:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762853.661196:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762853.661197:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762853.661200:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2828 00000100:00080000:0.0:1789762853.661203:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762853.661205:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762853.661208:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762853.661216:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b100 x1876699512993024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762853.661236:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762853/real 1789762853] req@ffff8dc005149880 x1876699512992640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762853.661252:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149880 x1876699512992640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762853.661266:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762853.661271:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762853.661276:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512992640:192.168.203.156@tcp:250:kworker.0 00010000:00010000:1.0:1789762853.661286:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762853.661300:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512992768:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762853.661324:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762853.661335:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762853/real 1789762853] req@ffff8dc00514a300 x1876699512992768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762853.661344:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514a300 x1876699512992768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762853.661350:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762853.661352:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762853.661354:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512992768:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762853.661358:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512992896:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762853.661372:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762853.661378:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762853/real 1789762853] req@ffff8dc005148e00 x1876699512992896/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762853.661386:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148e00 x1876699512992896/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762853.661396:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762853.661398:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762853.661400:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512992896:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762853.661404:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512993024:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762853.661413:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762853.661419:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762853/real 1789762853] req@ffff8dc00514b100 x1876699512993024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762853.661426:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b100 x1876699512993024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762853.661432:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762853.661434:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762853.661435:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512993024:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762854.685045:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762855.709200:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762856.733073:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762857.757048:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762858.781018:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762858.781029:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762858.781034:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1789762858.781034:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762858.781047:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2833 00000100:00080000:1.0:1789762858.781056:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762858.781075:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762858.781109:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512993152/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762858.781135:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762858.781139:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762858.781142:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762858.781147:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2833 00000100:00080000:1.0:1789762858.781152:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762858.781157:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762858.781162:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512993152:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1789762858.781169:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762858.781198:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512993280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762858.781211:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762858.781215:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762858.781217:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762858.781221:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2833 00000100:00080000:1.0:1789762858.781225:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789762858.781229:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789762858.781232:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762858.781240:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762858.781255:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512993408/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762858.781265:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762858/real 1789762858] req@ffff8dc005c94e00 x1876699512993152/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762858.781267:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762858.781277:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762858.781282:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762858.781289:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2833 00000100:00100000:2.0:1789762858.781290:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512993152/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762858.781295:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762858.781301:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762858.781304:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1789762858.781307:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762858.781313:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512993152:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1789762858.781314:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762858.781330:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512993536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762858.781335:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512993280:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762858.781358:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762858.781371:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762858/real 1789762858] req@ffff8dc005c97480 x1876699512993280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762858.781386:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512993280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762858.781395:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762858.781399:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762858.781402:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512993280:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762858.781409:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512993408:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762858.781428:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762858.781436:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762858/real 1789762858] req@ffff8dc005c96680 x1876699512993408/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762858.781444:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512993408/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762858.781461:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762858.781463:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762858.781465:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512993408:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762858.781475:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512993536:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762858.781485:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762858.781492:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762858/real 1789762858] req@ffff8dc005c95880 x1876699512993536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762883 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762858.781500:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512993536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762883 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762858.781505:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762858.781507:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762858.781509:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512993536:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762859.805068:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762860.829068:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762861.853067:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762862.877079:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762863.900987:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762863.901002:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762863.901008:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762863.901023:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2838 00000100:00080000:1.0:1789762863.901037:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762863.901047:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789762863.901052:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789762863.901096:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96d80 x1876699512993664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762863.901126:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762863.901129:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762863.901132:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762863.901137:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2838 00000100:00080000:1.0:1789762863.901141:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762863.901146:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762863.901157:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762863.901180:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699512993792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762863.901180:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512993664:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789762863.901208:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762863.901213:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762863.901216:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762863.901223:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2838 00000100:00080000:1.0:1789762863.901227:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762863.901231:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762863.901238:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762863.901254:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699512993920/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762863.901267:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789762863.901270:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762863.901276:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762863.901279:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762863.901287:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2838 00000100:00080000:1.0:1789762863.901293:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762863.901298:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762863.901305:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762863.901312:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762863/real 1789762863] req@ffff8dc005c96d80 x1876699512993664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762863.901323:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699512994048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762863.901333:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96d80 x1876699512993664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762863.901352:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762863.901357:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762863.901360:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512993664:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762863.901383:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512993792:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762863.901400:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762863.901408:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762863/real 1789762863] req@ffff8dc005c95880 x1876699512993792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762863.901417:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699512993792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762863.901423:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762863.901426:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762863.901428:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512993792:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762863.901433:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512993920:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762863.901444:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762863.901450:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762863/real 1789762863] req@ffff8dc005c96680 x1876699512993920/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762863.901458:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699512993920/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762863.901464:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762863.901465:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762863.901467:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512993920:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762863.901471:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512994048:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762863.901481:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762863.901488:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762863/real 1789762863] req@ffff8dc005c97480 x1876699512994048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762863.901495:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699512994048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762863.901500:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762863.901502:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762863.901505:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512994048:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762864.925009:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762865.949023:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762866.973024:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762867.997029:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762869.021036:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762869.021045:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762869.021048:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762869.021058:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2843 00000100:00080000:1.0:1789762869.021069:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762869.021076:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762869.021115:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699512994176/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1789762869.021130:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762869.021140:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762869.021145:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762869.021147:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762869.021150:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2843 00000100:00080000:1.0:1789762869.021154:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762869.021158:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762869.021167:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762869.021177:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699512994304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762869.021184:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512994176:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789762869.021187:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762869.021191:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762869.021192:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762869.021195:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2843 00000100:00080000:1.0:1789762869.021197:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762869.021199:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762869.021203:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762869.021211:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699512994432/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762869.021218:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762869.021221:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762869.021222:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762869.021225:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2843 00000100:00080000:1.0:1789762869.021227:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762869.021230:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762869.021234:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762869.021241:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699512994560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789762869.021245:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762869.021296:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762869/real 1789762869] req@ffff8dc005c94e00 x1876699512994176/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762869.021310:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699512994176/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762869.021318:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762869.021321:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762869.021324:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512994176:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762869.021342:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512994304:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762869.021358:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762869.021368:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762869/real 1789762869] req@ffff8dc005c95500 x1876699512994304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762869.021377:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699512994304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762869.021383:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762869.021385:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762869.021387:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512994304:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762869.021391:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512994432:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762869.021403:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762869.021409:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762869/real 1789762869] req@ffff8dc005c96300 x1876699512994432/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762869.021417:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699512994432/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762869.021422:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762869.021424:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762869.021426:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512994432:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762869.021430:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512994560:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762869.021440:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762869.021448:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762869/real 1789762869] req@ffff8dc005c97b80 x1876699512994560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762869.021460:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699512994560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762869.021468:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762869.021472:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762869.021475:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512994560:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762870.045058:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762871.069084:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762872.093070:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762873.117057:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762874.141012:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762874.141022:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762874.141027:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762874.141039:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2848 00000100:00080000:0.0:1789762874.141048:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762874.141056:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762874.141091:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149500 x1876699512994688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762874.141125:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762874.141129:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762874.141131:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762874.141136:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2848 00000100:00080000:0.0:1789762874.141141:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762874.141146:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762874.141154:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512994688:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1789762874.141155:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762874.141177:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b100 x1876699512994816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762874.141189:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762874.141193:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762874.141194:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762874.141198:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2848 00000100:00080000:0.0:1789762874.141203:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762874.141207:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762874.141213:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:2.0:1789762874.141214:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762874.141231:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148e00 x1876699512994944/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762874.141250:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789762874.141254:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762874/real 1789762874] req@ffff8dc005149500 x1876699512994688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762874.141256:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762874.141260:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762874.141270:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2848 00000100:00080000:0.0:1789762874.141275:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789762874.141275:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149500 x1876699512994688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762874.141285:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1789762874.141288:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1789762874.141293:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762874.141293:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762874.141297:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512994688:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762874.141311:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514a300 x1876699512995072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762874.141320:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512994816:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762874.141339:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762874.141349:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762874/real 1789762874] req@ffff8dc00514b100 x1876699512994816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762874.141358:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b100 x1876699512994816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762874.141365:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762874.141368:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762874.141371:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512994816:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762874.141377:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512994944:192.168.203.156@tcp:8:kworker.0 00010000:00010000:1.0:1789762874.141386:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.0:1789762874.141393:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762874.141402:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762874/real 1789762874] req@ffff8dc005148e00 x1876699512994944/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762874.141420:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148e00 x1876699512994944/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762874.141430:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762874.141433:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762874.141435:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512994944:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762874.141443:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512995072:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762874.141455:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762874.141462:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762874/real 1789762874] req@ffff8dc00514a300 x1876699512995072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762874.141484:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514a300 x1876699512995072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762874.141490:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762874.141492:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762874.141495:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512995072:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762875.165036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762876.189090:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762877.213007:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762878.237006:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762879.260996:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762879.261004:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762879.261007:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762879.261016:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2853 00000100:00080000:0.0:1789762879.261025:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762879.261031:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762879.261063:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149880 x1876699512995200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762879.261088:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762879.261091:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762879.261092:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762879.261096:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2853 00000100:00080000:0.0:1789762879.261099:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762879.261103:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762879.261110:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762879.261116:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512995200:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762879.261123:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149f80 x1876699512995328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762879.261131:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762879.261133:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762879.261134:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762879.261137:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2853 00000100:00080000:0.0:1789762879.261139:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762879.261142:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762879.261145:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762879.261152:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149180 x1876699512995456/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762879.261159:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762879.261161:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762879.261162:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762879.261165:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2853 00000100:00080000:0.0:1789762879.261168:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762879.261170:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762879.261173:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762879.261179:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b800 x1876699512995584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762879.261216:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512995328:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762879.261241:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512995456:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762879.261260:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512995584:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762879.261268:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762880.285033:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762881.309020:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762882.333105:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789762882.333182:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1789762882.333226:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789762882.333250:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789762882.333290:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789762882.333302:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789762882.333313:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1789762882.333355:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762879/real 1789762882] req@ffff8dc005149f80 x1876699512995328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762882.333399:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149f80 x1876699512995328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762882.333429:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762882.333436:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762882.333440:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512995328:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762882.333450:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762879/real 1789762882] req@ffff8dc005149180 x1876699512995456/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762882.333464:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149180 x1876699512995456/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762882.333472:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762882.333476:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762882.333480:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512995456:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762882.333488:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762879/real 1789762882] req@ffff8dc00514b800 x1876699512995584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762882.333501:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b800 x1876699512995584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762882.333510:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762882.333514:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762882.333518:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512995584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762882.333527:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762879/real 1789762882] req@ffff8dc005149880 x1876699512995200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762882.333541:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149880 x1876699512995200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762882.333556:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762882.333560:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762882.333563:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512995200:192.168.203.156@tcp:250:kworker.0 00010000:00010000:1.0:1789762883.357071:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762884.380999:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762884.381012:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762884.381018:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762884.381032:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2858 00000100:00080000:0.0:1789762884.381046:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762884.381060:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762884.381110:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149c00 x1876699512995712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762884.381146:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762884.381152:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762884.381156:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762884.381164:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2858 00000100:00080000:0.0:1789762884.381170:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762884.381177:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762884.381181:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512995712:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1789762884.381192:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762884.381213:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148380 x1876699512995840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762884.381227:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762884.381233:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762884.381235:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762884.381243:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2858 00000100:00080000:0.0:1789762884.381248:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762884.381255:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762884.381267:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1789762884.381275:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762884.381284:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148000 x1876699512995968/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762884.381294:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762884.381298:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762884.381301:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762884.381305:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2858 00000100:00080000:0.0:1789762884.381310:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762884.381314:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762884.381321:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762884.381330:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762884/real 1789762884] req@ffff8dc005149c00 x1876699512995712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762884.381337:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b480 x1876699512996096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762884.381364:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149c00 x1876699512995712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762884.381380:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762884.381386:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762884.381405:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512995712:192.168.203.156@tcp:250:kworker.0 00010000:00010000:0.0:1789762884.381423:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762884.381454:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512995840:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762884.381483:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762884.381499:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762884/real 1789762884] req@ffff8dc005148380 x1876699512995840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762884.381512:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148380 x1876699512995840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762884.381520:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762884.381524:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762884.381527:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512995840:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762884.381534:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512995968:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789762884.381555:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762884.381565:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762884/real 1789762884] req@ffff8dc005148000 x1876699512995968/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762884.381576:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148000 x1876699512995968/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762884.381584:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762884.381587:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762884.381590:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512995968:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789762884.381596:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512996096:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762884.381615:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762884.381627:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762884/real 1789762884] req@ffff8dc00514b480 x1876699512996096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762884.381654:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b480 x1876699512996096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762884.381662:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762884.381665:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762884.381669:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512996096:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762885.405030:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762886.429080:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762887.453029:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762888.477030:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762889.500991:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762889.501000:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762889.501004:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762889.501012:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2864 00000100:00080000:0.0:1789762889.501020:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762889.501026:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762889.501107:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514bb80 x1876699512996224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762889.501129:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762889.501133:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762889.501134:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762889.501137:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2864 00000100:00080000:0.0:1789762889.501140:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762889.501144:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762889.501152:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762889.501160:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512996224:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762889.501165:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514ad80 x1876699512996352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762889.501174:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762889.501176:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762889.501177:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762889.501180:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2864 00000100:00080000:0.0:1789762889.501182:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762889.501184:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762889.501188:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762889.501209:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148700 x1876699512996480/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762889.501217:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789762889.501221:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1789762889.501223:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762889.501225:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762889.501229:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2864 00000100:00080000:0.0:1789762889.501233:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762889.501236:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762889.501241:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762889.501251:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514aa00 x1876699512996608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762889.501263:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762889/real 1789762889] req@ffff8dc00514bb80 x1876699512996224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762889.501277:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514bb80 x1876699512996224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762889.501287:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762889.501292:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762889.501297:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512996224:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762889.501323:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512996352:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762889.501339:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:3.0:1789762889.501350:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762889.501375:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762889/real 1789762889] req@ffff8dc00514ad80 x1876699512996352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762889.501384:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514ad80 x1876699512996352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762889.501390:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762889.501393:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762889.501395:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512996352:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762889.501399:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512996480:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789762889.501414:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762889.501421:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762889/real 1789762889] req@ffff8dc005148700 x1876699512996480/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762889.501428:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148700 x1876699512996480/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762889.501434:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762889.501436:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762889.501438:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512996480:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789762889.501441:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512996608:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762889.501451:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762889.501456:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762889/real 1789762889] req@ffff8dc00514aa00 x1876699512996608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762889.501463:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514aa00 x1876699512996608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762889.501468:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762889.501470:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762889.501473:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512996608:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762890.525012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762891.549079:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762892.573117:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762893.597013:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762894.621079:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1789762894.621085:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789762894.621087:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789762894.621092:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762894.621101:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2869 00000100:00080000:1.0:1789762894.621111:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762894.621117:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762894.621149:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699512996736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762894.621176:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762894.621178:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762894.621180:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762894.621182:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2869 00000100:00080000:1.0:1789762894.621185:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762894.621187:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762894.621194:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762894.621203:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699512996864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762894.621210:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762894.621212:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789762894.621214:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762894.621216:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2869 00000100:00080000:1.0:1789762894.621221:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762894.621223:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762894.621226:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762894.621231:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699512996992/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789762894.621238:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789762894.621240:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762894.621242:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762894.621245:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2869 00000100:00080000:1.0:1789762894.621247:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789762894.621252:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789762894.621257:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762894.621265:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512996736:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762894.621276:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699512997120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762894.621387:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762894.621462:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762894/real 1789762894] req@ffff8dc005c94380 x1876699512996736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762894.621483:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699512996736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762894.621494:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762894.621499:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762894.621502:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512996736:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762894.621510:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512996864:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762894.621531:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762894.621546:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762894/real 1789762894] req@ffff8dc005c95f80 x1876699512996864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762894.621556:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699512996864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762894.621563:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762894.621565:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762894.621568:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512996864:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762894.621572:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512996992:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789762894.621589:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762894.621597:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762894/real 1789762894] req@ffff8dc005c94700 x1876699512996992/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762894.621606:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699512996992/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762894.621613:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762894.621616:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762894.621618:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512996992:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789762894.621774:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512997120:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762894.621828:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762894.621850:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762894/real 1789762894] req@ffff8dc005c96a00 x1876699512997120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762894.621864:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699512997120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762894.621871:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762894.621874:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762894.621877:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512997120:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762895.645095:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762896.669031:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762897.693082:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762898.717215:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762899.741012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762899.741036:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762899.741042:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762899.741046:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762899.741054:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2874 00000100:00080000:2.0:1789762899.741062:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762899.741067:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762899.741101:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699512997248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762899.741122:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762899.741126:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762899.741127:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762899.741130:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2874 00000100:00080000:2.0:1789762899.741133:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762899.741136:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762899.741144:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762899.741154:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512997248:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762899.741155:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699512997376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762899.741168:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762899.741170:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762899.741171:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762899.741174:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2874 00000100:00080000:2.0:1789762899.741176:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762899.741179:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762899.741183:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762899.741192:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699512997504/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762899.741199:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762899.741202:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762899.741203:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762899.741205:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2874 00000100:00080000:2.0:1789762899.741215:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762899.741218:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762899.741220:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789762899.741225:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762899.741231:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699512997632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762899.741268:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762899/real 1789762899] req@ffff8dc00368ca80 x1876699512997248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762899.741301:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699512997248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762899.741313:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762899.741317:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762899.741320:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512997248:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762899.741338:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512997376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762899.741355:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762899.741363:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762899/real 1789762899] req@ffff8dc00368ea00 x1876699512997376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762899.741371:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699512997376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762899.741377:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762899.741379:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762899.741381:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512997376:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762899.741384:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512997504:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789762899.741395:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762899.741401:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762899/real 1789762899] req@ffff8dc00368c000 x1876699512997504/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762899.741409:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699512997504/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762899.741419:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762899.741421:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762899.741423:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512997504:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789762899.741426:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512997632:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762899.741436:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762899.741441:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762899/real 1789762899] req@ffff8dc00368ce00 x1876699512997632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762924 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762899.741448:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699512997632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762924 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762899.741453:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762899.741455:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762899.741457:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512997632:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762900.765028:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762901.789070:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762902.813020:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762903.837056:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762904.861010:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762904.861018:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762904.861022:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762904.861031:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2879 00010000:00010000:3.0:1789762904.861040:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762904.861041:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762904.861049:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762904.861094:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699512997760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762904.861112:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762904.861116:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762904.861117:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762904.861121:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2879 00000100:00080000:2.0:1789762904.861124:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762904.861127:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762904.861134:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762904.861142:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512997760:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762904.861146:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699512997888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762904.861155:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762904.861158:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762904.861160:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762904.861163:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2879 00000100:00080000:2.0:1789762904.861165:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762904.861168:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762904.861172:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762904.861180:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512998016/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762904.861189:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762904.861192:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762904.861193:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762904.861195:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2879 00000100:00080000:2.0:1789762904.861198:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1789762904.861202:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762904.861203:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762904.861208:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762904.861217:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512998144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762904.861239:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762904/real 1789762904] req@ffff8dc00368f100 x1876699512997760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762904.861266:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699512997760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762904.861274:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762904.861277:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762904.861281:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512997760:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762904.861299:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512997888:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762904.861316:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762904.861326:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762904/real 1789762904] req@ffff8dc00368f480 x1876699512997888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762904.861334:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699512997888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762904.861340:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762904.861343:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762904.861345:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512997888:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762904.861348:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512998016:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789762904.861360:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762904.861366:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762904/real 1789762904] req@ffff8dc00368fb80 x1876699512998016/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762904.861374:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512998016/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762904.861379:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762904.861381:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762904.861383:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512998016:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789762904.861386:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512998144:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762904.861397:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762904.861404:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762904/real 1789762904] req@ffff8dc00368d880 x1876699512998144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762904.861411:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512998144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762904.861416:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762904.861418:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762904.861420:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512998144:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762905.885019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762906.909084:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762907.933007:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762908.957101:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789762909.981015:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762909.981024:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789762909.981028:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762909.981037:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2884 00000100:00080000:0.0:1789762909.981046:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762909.981065:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762909.981100:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148a80 x1876699512998272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762909.981121:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762909.981124:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762909.981125:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762909.981129:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2884 00000100:00080000:0.0:1789762909.981131:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762909.981134:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762909.981142:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762909.981166:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514aa00 x1876699512998400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789762909.981168:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512998272:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789762909.981177:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762909.981182:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789762909.981185:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762909.981188:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2884 00000100:00080000:0.0:1789762909.981190:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762909.981192:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762909.981195:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789762909.981216:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148700 x1876699512998528/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789762909.981238:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789762909.981240:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789762909.981242:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789762909.981246:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2884 00000100:00080000:0.0:1789762909.981249:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762909.981252:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789762909.981256:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762909.981266:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514ad80 x1876699512998656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789762909.981272:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762909.981323:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762909/real 1789762909] req@ffff8dc005148a80 x1876699512998272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762909.981341:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148a80 x1876699512998272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:2.0:1789762909.981342:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762909.981352:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762909.981357:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762909.981360:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512998272:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762909.981381:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512998400:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762909.981398:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762909.981406:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762909/real 1789762909] req@ffff8dc00514aa00 x1876699512998400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762909.981415:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514aa00 x1876699512998400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762909.981422:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762909.981424:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762909.981427:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512998400:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789762909.981433:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512998528:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789762909.981453:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762909.981464:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762909/real 1789762909] req@ffff8dc005148700 x1876699512998528/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762909.981487:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148700 x1876699512998528/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762909.981494:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762909.981496:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762909.981498:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512998528:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789762909.981502:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512998656:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789762909.981517:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789762909.981524:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762909/real 1789762909] req@ffff8dc00514ad80 x1876699512998656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762909.981531:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514ad80 x1876699512998656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762909.981536:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789762909.981538:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762909.981541:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512998656:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762911.005045:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762912.029037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762913.053269:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762914.077031:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762915.101047:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762915.101060:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762915.101066:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762915.101081:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2889 00010000:00010000:0.0:1789762915.101083:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762915.101103:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762915.101113:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762915.101164:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699512998784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762915.101211:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762915.101216:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762915.101220:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762915.101228:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2889 00000100:00080000:3.0:1789762915.101234:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762915.101242:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762915.101258:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762915.101259:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512998784:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762915.101286:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699512998912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762915.101299:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762915.101304:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762915.101306:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762915.101310:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2889 00000100:00080000:3.0:1789762915.101328:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762915.101334:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1789762915.101342:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789762915.101346:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762915.101364:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512999040/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762915.101377:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762915.101382:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762915.101383:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789762915.101389:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762915/real 1789762915] req@ffff8dc0107af800 x1876699512998784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789762915.101393:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2889 00000100:00080000:3.0:1789762915.101405:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762915.101414:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762915.101415:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699512998784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789762915.101428:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1789762915.101440:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762915.101446:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789762915.101450:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512999168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789762915.101463:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512998784:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762915.101498:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512998912:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762915.101538:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762915.101552:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762915/real 1789762915] req@ffff8dc0107ae680 x1876699512998912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762915.101566:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699512998912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762915.101575:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762915.101580:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762915.101583:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512998912:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789762915.101591:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512999040:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789762915.101614:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762915.101627:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762915/real 1789762915] req@ffff8dc0107ace00 x1876699512999040/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762915.101648:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512999040/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762915.101657:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762915.101660:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762915.101663:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512999040:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789762915.101667:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512999168:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789762915.101684:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762915.101693:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762915/real 1789762915] req@ffff8dc0107af100 x1876699512999168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762915.101706:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512999168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762915.101711:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762915.101713:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789762915.101715:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512999168:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762916.125021:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762917.149104:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762918.173138:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762919.197054:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762920.221098:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762920.221108:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762920.221112:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762920.221123:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2894 00000100:00080000:3.0:1789762920.221151:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762920.221160:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789762920.221182:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789762920.221211:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699512999296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762920.221235:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762920.221240:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762920.221243:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762920.221248:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2894 00000100:00080000:3.0:1789762920.221253:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762920.221257:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762920.221266:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762920.221277:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512999296:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762920.221281:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699512999424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762920.221303:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762920.221307:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762920.221310:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762920.221316:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2894 00000100:00080000:3.0:1789762920.221320:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762920.221323:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762920.221329:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762920.221340:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699512999552/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762920.221347:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762920.221350:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762920.221352:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762920.221355:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2894 00000100:00080000:3.0:1789762920.221357:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762920.221360:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762920.221364:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762920.221368:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762920.221405:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762920/real 1789762920] req@ffff8dc0107ad880 x1876699512999296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762920.221420:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699512999296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762920.221429:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762920.221434:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762920.221439:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512999296:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762920.221466:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512999424:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762920.221498:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762920.221513:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762920/real 1789762920] req@ffff8dc0107aea00 x1876699512999424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762920.221527:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699512999424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762920.221536:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762920.221540:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762920.221543:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512999424:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762920.221550:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512999552:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762920.221574:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762920.221586:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762920/real 1789762920] req@ffff8dc0107af100 x1876699512999552/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762920.221599:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699512999552/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762920.221614:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762920.221618:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762920.221622:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512999552:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789762920.221646:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699512999680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762920.221706:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512999680:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762920.221732:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762920.221744:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762920/real 1789762920] req@ffff8dc0107ace00 x1876699512999680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762920.221752:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699512999680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762920.221758:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762920.221761:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762920.221764:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699512999680:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762921.245033:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762922.269191:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762923.293035:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762924.317073:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762925.341038:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762925.341053:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762925.341063:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762925.341067:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762925.341079:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2899 00000100:00080000:2.0:1789762925.341089:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762925.341106:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762925.341156:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699512999808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762925.341178:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762925.341181:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762925.341182:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762925.341186:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2899 00000100:00080000:2.0:1789762925.341188:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762925.341190:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762925.341200:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762925.341212:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699512999936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762925.341220:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762925.341222:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762925.341224:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1789762925.341231:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512999808:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762925.341233:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2899 00000100:00080000:2.0:1789762925.341241:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762925.341249:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762925.341254:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762925.341265:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699513000064/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762925.341273:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762925.341276:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762925.341277:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762925.341280:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2899 00000100:00080000:2.0:1789762925.341282:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762925.341284:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762925.341287:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762925.341294:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699513000192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762925.341327:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762925.341368:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762925/real 1789762925] req@ffff8dc00368d880 x1876699512999808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762925.341390:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699512999808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762925.341402:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762925.341408:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762925.341412:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699512999808:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762925.341437:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512999936:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762925.341457:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762925.341467:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762925/real 1789762925] req@ffff8dc00368fb80 x1876699512999936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762925.341476:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699512999936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762925.341482:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762925.341485:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762925.341487:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699512999936:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762925.341491:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513000064:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762925.341503:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762925.341510:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762925/real 1789762925] req@ffff8dc00368f480 x1876699513000064/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762925.341518:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699513000064/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762925.341523:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762925.341525:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762925.341527:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513000064:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762925.341531:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513000192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762925.341541:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762925.341548:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762925/real 1789762925] req@ffff8dc00368f100 x1876699513000192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762925.341555:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699513000192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762925.341560:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762925.341562:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762925.341563:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513000192:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762926.365021:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762927.389019:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762928.413109:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762929.437037:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762930.460990:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762930.460997:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762930.461001:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762930.461010:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2905 00000100:00080000:2.0:1789762930.461018:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762930.461024:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762930.461075:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699513000320/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1789762930.461093:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762930.461097:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762930.461101:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762930.461102:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762930.461105:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2905 00000100:00080000:2.0:1789762930.461109:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762930.461113:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762930.461121:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762930.461133:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513000320:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762930.461134:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699513000448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762930.461147:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762930.461150:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762930.461151:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762930.461153:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2905 00000100:00080000:2.0:1789762930.461156:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762930.461158:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762930.461162:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762930.461170:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699513000576/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762930.461182:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762930.461184:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762930.461186:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762930.461189:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2905 00000100:00080000:2.0:1789762930.461191:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762930.461193:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762930.461196:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789762930.461199:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762930.461207:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699513000704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762930.461239:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762930/real 1789762930] req@ffff8dc00368ce00 x1876699513000320/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762930.461253:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699513000320/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762930.461261:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762930.461264:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762930.461267:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513000320:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762930.461284:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513000448:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762930.461298:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762930.461305:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762930/real 1789762930] req@ffff8dc00368c000 x1876699513000448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762930.461314:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699513000448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762930.461319:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762930.461321:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762930.461323:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513000448:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762930.461327:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513000576:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762930.461338:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762930.461344:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762930/real 1789762930] req@ffff8dc00368ea00 x1876699513000576/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762930.461351:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699513000576/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762930.461356:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762930.461358:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762930.461360:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513000576:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762930.461363:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513000704:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762930.461373:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762930.461382:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762930/real 1789762930] req@ffff8dc00368ca80 x1876699513000704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762930.461394:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699513000704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762930.461409:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762930.461412:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762930.461416:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513000704:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762931.485110:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762932.509146:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762933.533171:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762934.557267:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762935.581072:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762935.581089:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762935.581097:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762935.581116:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2910 00010000:00010000:3.0:1789762935.581120:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762935.581137:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762935.581153:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762935.581232:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699513000832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762935.581291:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762935.581300:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762935.581304:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762935.581314:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2910 00000100:00080000:2.0:1789762935.581326:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762935.581339:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762935.581355:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513000832:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789762935.581359:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762935.581404:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699513000960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762935.581432:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762935.581442:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1789762935.581449:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762935.581452:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762935.581469:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2910 00000100:00080000:2.0:1789762935.581481:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762935.581492:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762935.581508:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789762935.581509:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762935/real 1789762935] req@ffff8dc00368df80 x1876699513000832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762935.581555:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699513001088/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762935.581578:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699513000832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762935.581591:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762935.581612:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789762935.581616:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762935.581619:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789762935.581632:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762935.581644:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513000832:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762935.581662:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2910 00000100:00080000:2.0:1789762935.581679:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762935.581693:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789762935.581698:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513000960:192.168.203.156@tcp:38:kworker.0 00000008:00100000:2.0:1789762935.581711:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762935.581746:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699513001216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762935.581753:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762935.581782:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762935/real 1789762935] req@ffff8dc00368ed80 x1876699513000960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762935.581804:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699513000960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762935.581816:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762935.581822:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762935.581827:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513000960:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762935.581835:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513001088:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762935.581858:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762935.581870:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762935/real 1789762935] req@ffff8dc00368e300 x1876699513001088/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762935.581885:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699513001088/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762935.581959:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762935.581966:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762935.581974:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513001088:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762935.582002:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513001216:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762935.582076:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762935.582095:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762935/real 1789762935] req@ffff8dc00368f800 x1876699513001216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762935.582116:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699513001216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762935.582132:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762935.582138:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762935.582145:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513001216:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762936.605084:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762937.629135:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762938.653154:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762939.677110:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762940.701033:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762940.701041:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762940.701045:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789762940.701046:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762940.701056:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2915 00000100:00080000:3.0:1789762940.701066:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762940.701071:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762940.701106:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699513001344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762940.701127:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762940.701131:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762940.701132:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762940.701135:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2915 00000100:00080000:3.0:1789762940.701138:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762940.701141:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762940.701149:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762940.701160:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699513001472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762940.701169:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762940.701171:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513001344:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789762940.701173:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762940.701177:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762940.701185:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2915 00000100:00080000:3.0:1789762940.701187:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762940.701189:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762940.701193:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762940.701202:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699513001600/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762940.701209:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762940.701212:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762940.701213:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762940.701216:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2915 00000100:00080000:3.0:1789762940.701220:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762940.701223:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762940.701225:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762940.701232:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699513001728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789762940.701248:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762940.701299:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762940/real 1789762940] req@ffff8dc0107ae680 x1876699513001344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762940.701320:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699513001344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762940.701332:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762940.701336:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762940.701341:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513001344:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762940.701366:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513001472:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762940.701391:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762940.701404:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762940/real 1789762940] req@ffff8dc0107af800 x1876699513001472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762940.701417:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699513001472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762940.701426:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762940.701429:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762940.701433:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513001472:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762940.701440:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513001600:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762940.701460:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762940.701473:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762940/real 1789762940] req@ffff8dc0107ac380 x1876699513001600/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762940.701492:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699513001600/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762940.701501:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762940.701505:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762940.701509:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513001600:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762940.701515:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513001728:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762940.701534:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762940.701545:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762940/real 1789762940] req@ffff8dc0107adf80 x1876699513001728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762940.701557:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699513001728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762940.701566:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762940.701569:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762940.701573:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513001728:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789762941.725115:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762942.749093:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762943.773015:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762944.797094:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762945.820997:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762945.821008:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762945.821013:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762945.821025:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2920 00000100:00080000:2.0:1789762945.821035:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1789762945.821037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762945.821042:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762945.821077:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699513001856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762945.821100:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762945.821103:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762945.821104:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762945.821107:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2920 00000100:00080000:2.0:1789762945.821110:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762945.821113:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762945.821122:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762945.821131:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699513001984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762945.821140:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1789762945.821142:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513001856:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762945.821143:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762945.821145:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762945.821152:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2920 00000100:00080000:2.0:1789762945.821155:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762945.821157:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762945.821161:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762945.821170:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699513002112/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762945.821178:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762945.821181:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762945.821182:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762945.821185:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2920 00000100:00080000:2.0:1789762945.821187:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762945.821190:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762945.821193:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762945.821200:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699513002240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762945.821266:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513001984:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762945.821311:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513002112:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762945.821347:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513002240:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762946.845041:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762947.869058:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762948.893279:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789762948.893393:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:2.0:1789762948.893456:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789762948.893481:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789762948.893555:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789762948.893568:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789762948.893578:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1789762948.893801:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762945/real 1789762948] req@ffff8dc00368c700 x1876699513001984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762948.893839:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699513001984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762948.893857:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762948.893865:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762948.893871:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513001984:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762948.893883:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762945/real 1789762948] req@ffff8dc00368dc00 x1876699513002112/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762948.893938:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699513002112/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762948.893949:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762948.893954:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762948.893958:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513002112:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762948.893966:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762945/real 1789762948] req@ffff8dc00368e680 x1876699513002240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762948.893982:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699513002240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762948.893992:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762948.893996:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762948.894000:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513002240:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762948.894010:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762945/real 1789762948] req@ffff8dc00368d180 x1876699513001856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762948.894025:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699513001856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762948.894034:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762948.894038:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762948.894042:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513001856:192.168.203.156@tcp:250:kworker.0 00010000:00010000:0.0:1789762949.917027:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762950.940991:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762950.941000:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762950.941004:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762950.941012:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2925 00010000:00010000:3.0:1789762950.941021:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762950.941022:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762950.941030:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762950.941061:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699513002368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762950.941085:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762950.941089:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762950.941091:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762950.941096:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2925 00000100:00080000:2.0:1789762950.941100:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762950.941105:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762950.941116:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789762950.941117:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513002368:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762950.941138:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699513002496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762950.941150:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762950.941154:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762950.941156:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762950.941160:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2925 00000100:00080000:2.0:1789762950.941165:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762950.941169:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762950.941176:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1789762950.941186:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762950.941190:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699513002624/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762950.941199:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762950.941202:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762950.941204:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762950.941207:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2925 00000100:00080000:2.0:1789762950.941211:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762950.941213:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762950.941217:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762950.941229:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699513002752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789762950.941234:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762950/real 1789762950] req@ffff8dc00368d180 x1876699513002368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762950.941272:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699513002368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762950.941284:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762950.941289:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762950.941293:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513002368:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789762950.941321:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513002496:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762950.941346:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762950.941360:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762950/real 1789762950] req@ffff8dc00368e680 x1876699513002496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762950.941371:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699513002496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762950.941379:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762950.941383:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762950.941386:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513002496:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789762950.941392:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513002624:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789762950.941412:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762950.941423:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762950/real 1789762950] req@ffff8dc00368dc00 x1876699513002624/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762950.941434:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699513002624/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762950.941441:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762950.941445:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762950.941448:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513002624:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789762950.941454:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513002752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789762950.941472:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789762950.941483:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762950/real 1789762950] req@ffff8dc00368c700 x1876699513002752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789762950.941495:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699513002752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789762950.941503:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789762950.941506:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789762950.941509:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513002752:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762951.965129:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762952.989047:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762954.013070:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762955.037144:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762956.061019:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762956.061028:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762956.061032:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762956.061041:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2930 00000100:00080000:3.0:1789762956.061050:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762956.061056:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762956.061090:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699513002880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1789762956.061105:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762956.061111:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762956.061115:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762956.061117:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762956.061120:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2930 00000100:00080000:3.0:1789762956.061125:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762956.061131:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762956.061134:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513002880:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789762956.061140:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762956.061155:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699513003008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762956.061165:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762956.061168:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762956.061170:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762956.061175:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2930 00000100:00080000:3.0:1789762956.061180:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762956.061184:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762956.061187:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762956.061198:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699513003136/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762956.061206:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762956.061206:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762956.061212:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762956.061214:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762956.061220:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2930 00000100:00080000:3.0:1789762956.061223:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762956.061228:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762956.061232:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762956.061242:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699513003264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762956.061254:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762956/real 1789762956] req@ffff8dc0107af480 x1876699513002880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762956.061274:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699513002880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762956.061282:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762956.061284:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762956.061287:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513002880:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762956.061306:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513003008:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762956.061321:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762956.061331:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762956/real 1789762956] req@ffff8dc0107ad180 x1876699513003008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762956.061340:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699513003008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762956.061346:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762956.061349:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762956.061352:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513003008:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762956.061357:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513003136:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762956.061373:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762956.061381:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762956/real 1789762956] req@ffff8dc0107ac700 x1876699513003136/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762956.061391:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699513003136/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762956.061400:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762956.061403:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762956.061405:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513003136:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762956.061409:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513003264:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762956.061420:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762956.061427:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762956/real 1789762956] req@ffff8dc0107aed80 x1876699513003264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762956.061433:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699513003264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762956.061438:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762956.061440:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762956.061442:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513003264:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762957.085044:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762958.109087:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762959.133071:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762960.157092:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762961.181005:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762961.181019:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762961.181023:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762961.181035:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2935 00000100:00080000:3.0:1789762961.181045:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1789762961.181052:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762961.181052:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762961.181110:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699513003392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762961.181135:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762961.181137:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762961.181139:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762961.181142:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2935 00000100:00080000:3.0:1789762961.181146:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762961.181150:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762961.181160:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762961.181170:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513003392:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762961.181175:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699513003520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762961.181185:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762961.181187:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762961.181188:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762961.181190:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2935 00000100:00080000:3.0:1789762961.181192:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762961.181195:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762961.181198:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762961.181207:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699513003648/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762961.181214:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762961.181217:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762961.181218:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762961.181221:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2935 00000100:00080000:3.0:1789762961.181224:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762961.181227:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762961.181238:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789762961.181251:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699513003776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762961.181263:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762961.181306:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762961/real 1789762961] req@ffff8dc0107ad500 x1876699513003392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762961.181322:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699513003392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762961.181331:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762961.181335:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762961.181338:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513003392:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762961.181359:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513003520:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762961.181374:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762961.181382:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762961/real 1789762961] req@ffff8dc0107ac000 x1876699513003520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762961.181391:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699513003520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762961.181396:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762961.181399:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762961.181401:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513003520:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762961.181404:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513003648:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762961.181415:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762961.181421:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762961/real 1789762961] req@ffff8dc0107ae300 x1876699513003648/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762961.181428:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699513003648/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762961.181439:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762961.181443:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762961.181447:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513003648:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762961.181453:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513003776:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762961.181473:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762961.181485:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762961/real 1789762961] req@ffff8dc0107aca80 x1876699513003776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762961.181498:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699513003776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762961.181504:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762961.181506:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762961.181508:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513003776:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762962.205031:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762963.229051:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762964.253058:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762965.277036:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762966.301076:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1789762966.301088:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789762966.301100:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789762966.301109:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762966.301129:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2940 00000100:00080000:3.0:1789762966.301149:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762966.301162:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789762966.301231:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699513003904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762966.301269:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762966.301274:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762966.301277:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762966.301283:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2940 00000100:00080000:3.0:1789762966.301289:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762966.301296:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762966.301310:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762966.301333:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513003904:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789762966.301334:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699513004032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762966.301360:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789762966.301366:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789762966.301370:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762966.301377:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2940 00000100:00080000:3.0:1789762966.301384:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762966.301393:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762966.301403:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789762966.301434:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699513004160/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789762966.301457:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789762966.301466:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789762966.301471:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789762966.301478:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789762966.301490:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2940 00000100:00080000:3.0:1789762966.301501:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789762966.301507:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789762966.301516:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762966.301532:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762966/real 1789762966] req@ffff8dc0107adc00 x1876699513003904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789762966.301541:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699513004288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762966.301620:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699513003904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762966.301658:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762966.301664:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762966.301670:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513003904:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762966.301704:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513004032:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762966.301753:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762966.301771:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762966/real 1789762966] req@ffff8dc0107afb80 x1876699513004032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762966.301809:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699513004032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762966.301826:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762966.301830:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762966.301835:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513004032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762966.301844:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513004160:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762966.301886:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762966.301940:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762966/real 1789762966] req@ffff8dc0107aca80 x1876699513004160/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762966.301957:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699513004160/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762966.301967:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762966.301971:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762966.301975:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513004160:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762966.301984:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513004288:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762966.302010:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762966.302023:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762966/real 1789762966] req@ffff8dc0107ae300 x1876699513004288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762966.302038:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699513004288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762966.302048:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762966.302052:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762966.302056:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513004288:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789762967.325095:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762968.349011:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762969.373078:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762970.397103:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762971.421056:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762971.421065:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:0.0:1789762971.421072:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762971.421072:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762971.421085:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2946 00000100:00080000:2.0:1789762971.421096:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762971.421102:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762971.421137:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699513004416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762971.421164:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762971.421169:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762971.421171:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762971.421177:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2946 00000100:00080000:2.0:1789762971.421182:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762971.421187:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762971.421197:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762971.421205:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513004416:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762971.421219:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699513004544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762971.421234:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762971.421237:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762971.421239:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762971.421242:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2946 00000100:00080000:2.0:1789762971.421245:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762971.421248:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762971.421252:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762971.421264:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699513004672/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762971.421272:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762971.421275:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762971.421277:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762971.421281:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2946 00000100:00080000:2.0:1789762971.421285:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762971.421290:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762971.421296:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762971.421311:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762971.421313:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699513004800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762971.421369:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762971/real 1789762971] req@ffff8dc00368f800 x1876699513004416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762971.421391:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699513004416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789762996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762971.421403:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762971.421407:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762971.421412:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513004416:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762971.421441:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513004544:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762971.421469:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762971.421483:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762971/real 1789762971] req@ffff8dc00368e300 x1876699513004544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762971.421492:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699513004544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762971.421499:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762971.421501:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762971.421504:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513004544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762971.421509:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513004672:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762971.421531:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762971.421544:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762971/real 1789762971] req@ffff8dc00368ed80 x1876699513004672/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762971.421557:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699513004672/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789762996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762971.421567:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762971.421570:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762971.421573:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513004672:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762971.421578:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513004800:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762971.421595:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762971.421604:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762971/real 1789762971] req@ffff8dc00368df80 x1876699513004800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762971.421617:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699513004800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789762996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762971.421622:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762971.421625:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762971.421627:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513004800:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762972.445170:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762973.469103:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762974.493119:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762975.516986:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762976.541003:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762976.541014:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762976.541018:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762976.541028:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2951 00000100:00080000:2.0:1789762976.541041:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1789762976.541045:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762976.541050:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762976.541086:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699513004928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762976.541106:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762976.541111:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762976.541113:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762976.541117:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2951 00000100:00080000:2.0:1789762976.541121:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762976.541125:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762976.541135:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762976.541143:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513004928:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762976.541147:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699513005056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762976.541156:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762976.541159:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762976.541161:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762976.541164:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2951 00000100:00080000:2.0:1789762976.541167:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762976.541170:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762976.541175:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762976.541184:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699513005184/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762976.541192:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762976.541196:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762976.541197:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762976.541200:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2951 00000800:00000100:0.0:1789762976.541203:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762976.541203:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762976.541207:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762976.541211:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762976.541219:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699513005312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762976.541241:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762976/real 1789762976] req@ffff8dc00368ca80 x1876699513004928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762976.541258:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699513004928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762976.541269:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762976.541272:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762976.541275:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513004928:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762976.541295:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513005056:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762976.541312:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762976.541321:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762976/real 1789762976] req@ffff8dc00368ea00 x1876699513005056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762976.541335:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699513005056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762976.541340:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762976.541342:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762976.541344:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513005056:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762976.541348:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513005184:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762976.541359:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762976.541365:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762976/real 1789762976] req@ffff8dc00368c000 x1876699513005184/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762976.541373:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699513005184/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762976.541378:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762976.541380:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762976.541382:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513005184:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762976.541385:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513005312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762976.541395:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762976.541400:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762976/real 1789762976] req@ffff8dc00368ce00 x1876699513005312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762976.541407:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699513005312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762976.541415:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762976.541417:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762976.541419:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513005312:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762977.565030:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762978.589011:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762979.613120:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789762980.637073:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762981.661068:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762981.661081:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762981.661088:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762981.661103:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2956 00000100:00080000:2.0:1789762981.661117:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762981.661141:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789762981.661170:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762981.661194:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699513005440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762981.661238:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762981.661243:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762981.661246:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762981.661252:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2956 00000100:00080000:2.0:1789762981.661257:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762981.661264:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762981.661277:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762981.661290:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513005440:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762981.661310:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699513005568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762981.661339:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762981.661347:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762981.661351:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762981.661361:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2956 00000100:00080000:2.0:1789762981.661371:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789762981.661383:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762981.661385:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762981.661401:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762981.661435:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699513005696/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762981.661445:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762981/real 1789762981] req@ffff8dc00368f100 x1876699513005440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762981.661491:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789762981.661501:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699513005440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789762981.661502:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762981.661511:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762981.661527:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2956 00000100:00080000:0.0:1789762981.661537:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789762981.661540:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789762981.661550:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1789762981.661556:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762981.661566:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513005440:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789762981.661573:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762981.661598:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699513005824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762981.661607:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513005568:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762981.661654:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762981.661674:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762981/real 1789762981] req@ffff8dc00368f480 x1876699513005568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762981.661719:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699513005568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762981.661739:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762981.661744:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762981.661748:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513005568:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762981.661756:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513005696:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762981.661781:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762981.661795:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762981/real 1789762981] req@ffff8dc00368fb80 x1876699513005696/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762981.661809:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699513005696/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762981.661820:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762981.661824:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762981.661828:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513005696:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762981.661844:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513005824:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762981.661864:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762981.661876:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762981/real 1789762981] req@ffff8dc00368d880 x1876699513005824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762981.661890:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699513005824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762981.661933:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762981.661937:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762981.661942:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513005824:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762982.685056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789762983.709050:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762984.733074:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762985.757071:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762986.781078:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762986.781086:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762986.781089:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762986.781098:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2961 00000100:00080000:2.0:1789762986.781108:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762986.781114:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789762986.781127:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762986.781151:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699513005952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762986.781174:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762986.781192:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762986.781194:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762986.781198:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2961 00000100:00080000:2.0:1789762986.781201:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762986.781204:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762986.781212:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762986.781222:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699513006080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762986.781230:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513005952:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789762986.781245:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762986.781249:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762986.781251:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762986.781254:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2961 00000100:00080000:2.0:1789762986.781258:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762986.781260:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762986.781264:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762986.781273:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699513006208/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762986.781280:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762986.781283:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762986.781285:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762986.781290:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2961 00000100:00080000:2.0:1789762986.781295:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762986.781301:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762986.781307:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789762986.781318:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789762986.781345:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699513006336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762986.781379:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762986/real 1789762986] req@ffff8dc00368d880 x1876699513005952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762986.781406:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699513005952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762986.781416:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762986.781420:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762986.781424:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513005952:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762986.781448:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513006080:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762986.781463:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762986.781472:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762986/real 1789762986] req@ffff8dc00368fb80 x1876699513006080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762986.781480:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699513006080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762986.781486:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762986.781488:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762986.781490:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513006080:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762986.781494:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513006208:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762986.781508:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762986.781515:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762986/real 1789762986] req@ffff8dc00368f480 x1876699513006208/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762986.781523:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699513006208/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762986.781528:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762986.781530:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762986.781532:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513006208:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762986.781535:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513006336:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762986.781547:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762986.781553:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762986/real 1789762986] req@ffff8dc00368f100 x1876699513006336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763011 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762986.781560:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699513006336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763011 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762986.781565:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762986.781567:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762986.781569:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513006336:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789762987.805103:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762988.829142:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762989.853038:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762990.877012:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762991.901123:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762991.901138:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762991.901144:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762991.901160:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2966 00000100:00080000:2.0:1789762991.901177:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762991.901189:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789762991.901245:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699513006464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762991.901302:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762991.901308:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762991.901311:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762991.901322:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2966 00000100:00080000:2.0:1789762991.901328:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762991.901335:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762991.901351:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762991.901381:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513006464:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789762991.901382:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699513006592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762991.901425:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762991.901432:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762991.901443:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1789762991.901445:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762991.901464:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2966 00000100:00080000:2.0:1789762991.901473:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762991.901481:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762991.901492:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762991.901512:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699513006720/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762991.901529:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762991.901550:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762991.901557:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762991.901561:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762991.901572:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2966 00000100:00080000:2.0:1789762991.901577:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762991.901583:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762991.901592:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789762991.901633:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762991/real 1789762991] req@ffff8dc00368ce00 x1876699513006464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789762991.901673:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699513006848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762991.901678:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699513006464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762991.901755:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762991.901764:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762991.901773:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513006464:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762991.901818:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513006592:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762991.901855:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762991.901875:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762991/real 1789762991] req@ffff8dc00368c000 x1876699513006592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762991.901895:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699513006592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762991.901955:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762991.901960:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762991.901972:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513006592:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762991.901984:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513006720:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762991.902028:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762991.902047:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762991/real 1789762991] req@ffff8dc00368ea00 x1876699513006720/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762991.902069:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699513006720/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762991.902100:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762991.902107:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762991.902117:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513006720:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762991.902144:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513006848:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762991.902190:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762991.902214:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762991/real 1789762991] req@ffff8dc00368ca80 x1876699513006848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762991.902239:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699513006848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762991.902251:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762991.902255:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762991.902259:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513006848:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762992.925130:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789762993.948983:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762994.973030:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762995.997078:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789762997.021057:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762997.021067:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789762997.021072:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762997.021083:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2971 00000100:00080000:2.0:1789762997.021095:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762997.021102:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789762997.021128:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789762997.021171:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699513006976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762997.021202:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762997.021208:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762997.021210:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762997.021216:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2971 00000100:00080000:2.0:1789762997.021222:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762997.021227:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789762997.021236:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513006976:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789762997.021237:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762997.021256:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699513007104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789762997.021266:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762997.021269:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789762997.021271:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762997.021274:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2971 00000100:00080000:2.0:1789762997.021278:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762997.021281:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762997.021285:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789762997.021294:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699513007232/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789762997.021304:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789762997.021307:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789762997.021312:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789762997.021314:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789762997.021319:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2971 00000100:00080000:2.0:1789762997.021323:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789762997.021326:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789762997.021329:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789762997.021341:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699513007360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789762997.021350:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762997/real 1789762997] req@ffff8dc00368df80 x1876699513006976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762997.021398:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699513006976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762997.021415:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762997.021420:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762997.021424:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513006976:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789762997.021451:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513007104:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762997.021478:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762997.021491:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762997/real 1789762997] req@ffff8dc00368ed80 x1876699513007104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762997.021504:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699513007104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762997.021512:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762997.021516:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762997.021519:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513007104:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789762997.021525:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513007232:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789762997.021542:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762997.021551:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762997/real 1789762997] req@ffff8dc00368e300 x1876699513007232/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762997.021580:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699513007232/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762997.021606:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762997.021609:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762997.021611:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513007232:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789762997.021616:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513007360:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789762997.021633:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789762997.021642:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789762997/real 1789762997] req@ffff8dc00368f800 x1876699513007360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789762997.021650:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699513007360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789762997.021675:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789762997.021678:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789762997.021681:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513007360:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789762998.045062:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789762999.069082:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763000.093022:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763001.117065:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763002.141002:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763002.141019:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763002.141025:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763002.141036:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2976 00000100:00080000:3.0:1789763002.141048:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1789763002.141056:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763002.141056:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763002.141103:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699513007488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763002.141131:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763002.141135:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763002.141137:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763002.141141:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2976 00000100:00080000:3.0:1789763002.141144:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763002.141148:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763002.141157:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763002.141164:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513007488:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763002.141172:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699513007616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763002.141181:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763002.141184:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763002.141191:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763002.141195:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2976 00000100:00080000:3.0:1789763002.141197:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763002.141200:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763002.141204:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763002.141214:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699513007744/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763002.141222:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763002.141225:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763002.141226:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763002.141228:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2976 00000100:00080000:3.0:1789763002.141230:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763002.141233:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763002.141236:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763002.141245:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699513007872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763002.141253:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763002.141307:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763002/real 1789763002] req@ffff8dc0107ac000 x1876699513007488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763002.141324:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699513007488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763002.141333:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763002.141337:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763002.141340:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513007488:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763002.141363:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513007616:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763002.141378:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763002.141387:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763002/real 1789763002] req@ffff8dc0107ad500 x1876699513007616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763002.141401:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699513007616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763002.141407:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763002.141409:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763002.141412:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513007616:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763002.141415:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513007744:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763002.141426:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763002.141432:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763002/real 1789763002] req@ffff8dc0107aed80 x1876699513007744/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763002.141440:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699513007744/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763002.141446:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763002.141447:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763002.141449:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513007744:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763002.141453:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513007872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763002.141463:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763002.141469:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763002/real 1789763002] req@ffff8dc0107ac700 x1876699513007872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763002.141477:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699513007872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763002.141486:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763002.141488:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763002.141491:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513007872:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763003.165017:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763004.189060:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763005.213092:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763006.237026:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763007.261037:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763007.261046:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763007.261050:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763007.261062:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2981 00010000:00010000:3.0:1789763007.261063:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763007.261071:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763007.261078:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763007.261111:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699513008000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763007.261133:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763007.261135:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763007.261137:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763007.261139:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2981 00000100:00080000:1.0:1789763007.261142:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763007.261145:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763007.261153:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763007.261166:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699513008128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763007.261167:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513008000:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789763007.261183:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763007.261187:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763007.261190:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763007.261195:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2981 00000100:00080000:1.0:1789763007.261200:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763007.261205:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763007.261211:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789763007.261227:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763007.261227:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699513008256/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763007.261242:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763007.261247:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763007.261249:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763007.261255:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2981 00000100:00080000:1.0:1789763007.261260:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763007.261266:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763007.261270:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763007/real 1789763007] req@ffff8dc005c97800 x1876699513008000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789763007.261273:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763007.261288:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699513008000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763007.261291:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699513008384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763007.261308:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763007.261314:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763007.261319:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513008000:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763007.261339:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513008128:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763007.261353:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763007.261361:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763007/real 1789763007] req@ffff8dc005c95c00 x1876699513008128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763007.261369:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699513008128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763007.261375:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763007.261377:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763007.261380:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513008128:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763007.261383:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513008256:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763007.261394:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763007.261400:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763007/real 1789763007] req@ffff8dc005c94a80 x1876699513008256/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763007.261408:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699513008256/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763007.261413:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763007.261415:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763007.261416:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513008256:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763007.261420:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513008384:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763007.261430:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763007.261436:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763007/real 1789763007] req@ffff8dc005c97100 x1876699513008384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763007.261443:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699513008384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763007.261448:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763007.261450:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763007.261452:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513008384:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763008.285056:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763009.308997:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763010.333352:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763011.357031:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763012.380983:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763012.380991:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:0.0:1789763012.380993:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763012.380996:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763012.381005:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2986 00000100:00080000:2.0:1789763012.381014:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763012.381020:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763012.381052:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699513008512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763012.381080:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763012.381084:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763012.381086:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763012.381092:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2986 00000100:00080000:2.0:1789763012.381097:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763012.381103:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763012.381112:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763012.381120:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513008512:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763012.381128:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699513008640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763012.381138:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763012.381140:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763012.381142:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763012.381145:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2986 00000100:00080000:2.0:1789763012.381148:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763012.381150:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763012.381154:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763012.381163:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699513008768/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763012.381170:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763012.381173:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763012.381175:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763012.381177:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2986 00000100:00080000:2.0:1789763012.381179:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763012.381182:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763012.381184:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763012.381191:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699513008896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763012.381225:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513008640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763012.381255:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513008768:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763012.381275:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513008896:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763013.405021:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763014.429077:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763015.453087:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789763015.453155:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:1.0:1789763015.453202:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789763015.453219:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789763015.453265:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789763015.453271:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789763015.453277:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:0.0:1789763015.453387:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763012/real 1789763015] req@ffff8dc00368dc00 x1876699513008640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763015.453470:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699513008640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763015.453495:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763015.453505:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763015.453513:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513008640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763015.453530:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763012/real 1789763015] req@ffff8dc00368e680 x1876699513008768/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763015.453558:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699513008768/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763015.453574:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763015.453583:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763015.453591:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513008768:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763015.453607:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763012/real 1789763015] req@ffff8dc00368d180 x1876699513008896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763015.453635:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699513008896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763015.453652:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763015.453659:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763015.453666:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513008896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763015.453681:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763012/real 1789763015] req@ffff8dc00368c700 x1876699513008512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763015.453706:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699513008512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763015.453721:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763015.453727:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763015.453734:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513008512:192.168.203.156@tcp:250:kworker.0 00010000:00010000:3.0:1789763016.477099:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763017.501074:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763017.501087:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763017.501093:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763017.501113:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2992 00000100:00080000:2.0:1789763017.501130:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763017.501141:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763017.501197:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699513009024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763017.501234:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1789763017.501239:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763017.501241:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763017.501251:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763017.501261:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2992 00000100:00080000:2.0:1789763017.501267:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763017.501276:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763017.501290:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763017.501310:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699513009152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763017.501328:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513009024:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763017.501334:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763017.501356:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763017.501360:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763017.501373:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2992 00000100:00080000:2.0:1789763017.501385:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763017.501394:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763017.501407:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763017.501468:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699513009280/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763017.501491:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789763017.501504:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763017.501515:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763017.501520:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763017.501533:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2992 00000100:00080000:2.0:1789763017.501546:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763017.501556:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763017.501571:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763017/real 1789763017] req@ffff8dc00368c700 x1876699513009024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789763017.501577:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763017.501625:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699513009408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763017.501633:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699513009024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763017.501715:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763017.501728:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763017.501739:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513009024:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763017.501795:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513009152:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763017.501847:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763017.501876:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763017/real 1789763017] req@ffff8dc00368d180 x1876699513009152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763017.501948:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699513009152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763017.501966:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763017.501973:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763017.501980:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513009152:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763017.501993:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513009280:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763017.502036:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763017.502059:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763017/real 1789763017] req@ffff8dc00368e680 x1876699513009280/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763017.502081:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699513009280/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763017.502096:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763017.502103:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763017.502109:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513009280:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763017.502122:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513009408:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763017.502161:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763017.502184:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763017/real 1789763017] req@ffff8dc00368dc00 x1876699513009408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763017.502210:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699513009408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763017.502225:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763017.502232:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763017.502238:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513009408:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763018.525311:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763019.549140:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763020.573191:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763021.597166:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763022.621123:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763022.621140:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763022.621147:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1789763022.621161:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763022.621166:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 2997 00000100:00080000:1.0:1789763022.621186:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763022.621198:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763022.621264:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699513009536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763022.621305:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763022.621310:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763022.621312:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763022.621318:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2997 00000100:00080000:1.0:1789763022.621323:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763022.621329:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763022.621344:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763022.621365:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699513009664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763022.621403:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513009536:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789763022.621436:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763022.621453:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763022.621457:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763022.621469:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 2997 00000100:00080000:1.0:1789763022.621479:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763022.621488:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763022.621501:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763022.621528:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699513009792/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763022.621544:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763022.621556:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1789763022.621557:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789763022.621566:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763022.621577:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 2997 00000100:00080000:1.0:1789763022.621588:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763022.621595:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763022.621606:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763022.621626:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699513009920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763022.621667:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763022/real 1789763022] req@ffff8dc005c95180 x1876699513009536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763022.621713:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699513009536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763022.621730:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763022.621736:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763022.621741:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513009536:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763022.621788:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513009664:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763022.621827:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763022.621846:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763022/real 1789763022] req@ffff8dc005c97100 x1876699513009664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763022.621862:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699513009664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763022.621883:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763022.621887:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763022.621893:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513009664:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763022.621949:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513009792:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763022.621996:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763022.622020:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763022/real 1789763022] req@ffff8dc005c94a80 x1876699513009792/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763022.622046:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699513009792/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763022.622064:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763022.622071:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763022.622079:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513009792:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763022.622093:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513009920:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763022.622135:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763022.622152:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763022/real 1789763022] req@ffff8dc005c95c00 x1876699513009920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763022.622168:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699513009920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763022.622179:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763022.622183:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763022.622188:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513009920:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763023.645175:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763024.669171:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763025.693115:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763026.717127:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763027.741164:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763027.741178:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763027.741186:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763027.741205:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3002 00010000:00010000:3.0:1789763027.741213:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763027.741223:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763027.741237:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763027.741302:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699513010048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763027.741360:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763027.741369:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763027.741391:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763027.741398:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3002 00000100:00080000:2.0:1789763027.741408:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763027.741417:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763027.741431:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513010048:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789763027.741435:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763027.741478:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699513010176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763027.741499:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763027.741504:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763027.741507:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763027.741514:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3002 00000100:00080000:2.0:1789763027.741520:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763027.741525:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763027.741532:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763027.741548:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699513010304/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763027.741558:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763027.741562:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763027.741569:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763027.741571:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789763027.741578:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3002 00000100:00080000:2.0:1789763027.741588:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763027.741594:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763027.741601:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763027.741614:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699513010432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763027.741653:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763027/real 1789763027] req@ffff8dc00368f800 x1876699513010048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763027.741689:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699513010048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763027.741705:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763027.741711:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763027.741717:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513010048:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763027.741756:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513010176:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763027.741791:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763027.741808:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763027/real 1789763027] req@ffff8dc00368e300 x1876699513010176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763027.741827:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699513010176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763027.741839:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763027.741843:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763027.741847:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513010176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763027.741869:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513010304:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763027.741946:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763027.741969:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763027/real 1789763027] req@ffff8dc00368ed80 x1876699513010304/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763027.741987:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699513010304/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763027.741999:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763027.742005:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763027.742011:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513010304:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763027.742024:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513010432:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763027.742061:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763027.742100:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763027/real 1789763027] req@ffff8dc00368df80 x1876699513010432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763052 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763027.742127:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699513010432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763052 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763027.742165:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763027.742175:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763027.742181:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513010432:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763028.765212:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763029.789164:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763030.813066:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763031.837180:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763032.861121:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763032.861137:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763032.861146:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763032.861166:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3007 00000100:00080000:2.0:1789763032.861182:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1789763032.861194:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763032.861195:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763032.861281:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699513010560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763032.861348:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763032.861357:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763032.861361:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763032.861371:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3007 00000100:00080000:2.0:1789763032.861380:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763032.861391:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763032.861412:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763032.861436:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513010560:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763032.861442:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699513010688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763032.861473:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763032.861482:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763032.861486:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763032.861494:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3007 00000100:00080000:2.0:1789763032.861503:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763032.861512:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763032.861524:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763032.861552:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699513010816/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763032.861574:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763032.861581:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763032.861585:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763032.861597:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789763032.861602:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3007 00000100:00080000:2.0:1789763032.861615:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763032.861624:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763032.861634:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763032.861655:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699513010944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763032.861706:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763032/real 1789763032] req@ffff8dc00368ca80 x1876699513010560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763032.861748:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699513010560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763032.861771:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763032.861779:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763032.861788:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513010560:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763032.861831:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513010688:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763032.861878:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763032.861950:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763032/real 1789763032] req@ffff8dc00368ea00 x1876699513010688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763032.861979:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699513010688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763032.861995:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763032.862001:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763032.862009:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513010688:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763032.862021:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513010816:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763032.862060:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763032.862080:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763032/real 1789763032] req@ffff8dc00368c000 x1876699513010816/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763032.862101:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699513010816/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763032.862116:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763032.862121:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763032.862128:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513010816:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763032.862139:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513010944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763032.862173:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763032.862193:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763032/real 1789763032] req@ffff8dc00368ce00 x1876699513010944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763032.862215:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699513010944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763032.862229:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763032.862235:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763032.862241:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513010944:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763033.885062:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763034.909162:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763035.933043:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763036.957099:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763037.981097:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763037.981114:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763037.981121:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763037.981138:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3012 00010000:00010000:3.0:1789763037.981148:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763037.981157:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763037.981172:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763037.981239:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699513011072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763037.981312:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763037.981320:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763037.981324:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763037.981333:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3012 00000100:00080000:2.0:1789763037.981342:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763037.981352:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763037.981372:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763037.981400:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513011072:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763037.981401:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699513011200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763037.981448:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763037.981452:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763037.981456:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763037.981464:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3012 00000100:00080000:2.0:1789763037.981471:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763037.981476:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763037.981487:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763037.981503:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699513011328/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763037.981517:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763037.981523:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763037.981526:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789763037.981534:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789763037.981536:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3012 00000100:00080000:2.0:1789763037.981546:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763037.981553:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763037.981564:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763037.981580:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699513011456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763037.981604:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763037/real 1789763037] req@ffff8dc00368f100 x1876699513011072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763037.981636:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699513011072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763037.981653:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763037.981661:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763037.981670:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513011072:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763037.981714:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513011200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763037.981768:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763037.981789:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763037/real 1789763037] req@ffff8dc00368f480 x1876699513011200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763037.981806:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699513011200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763037.981818:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763037.981822:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763037.981827:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513011200:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763037.981836:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513011328:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763037.981865:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763037.981878:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763037/real 1789763037] req@ffff8dc00368fb80 x1876699513011328/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763037.981937:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699513011328/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763037.981949:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763037.981953:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763037.981957:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513011328:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763037.981964:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513011456:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763037.981991:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763037.982004:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763037/real 1789763037] req@ffff8dc00368d880 x1876699513011456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763037.982019:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699513011456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763037.982029:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763037.982033:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763037.982037:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513011456:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763039.005024:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763040.029179:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763041.053221:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763042.077356:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763043.100979:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763043.100987:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763043.100991:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763043.101001:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3017 00000100:00080000:0.0:1789763043.101012:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763043.101020:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763043.101056:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514bb80 x1876699513011584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763043.101078:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763043.101081:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763043.101083:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763043.101087:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3017 00000100:00080000:0.0:1789763043.101091:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763043.101097:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763043.101108:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763043.101117:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b480 x1876699513011712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763043.101127:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763043.101132:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1789763043.101134:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513011584:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789763043.101135:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763043.101157:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3017 00000100:00080000:0.0:1789763043.101160:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763043.101165:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763043.101169:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763043.101180:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148000 x1876699513011840/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763043.101187:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763043.101189:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763043.101191:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763043.101193:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3017 00000100:00080000:0.0:1789763043.101196:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763043.101200:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763043.101202:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763043.101210:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148380 x1876699513011968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763043.101285:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00010000:00010000:3.0:1789763043.101303:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763043.101362:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763043/real 1789763043] req@ffff8dc00514bb80 x1876699513011584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763043.101390:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514bb80 x1876699513011584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763043.101406:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763043.101411:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763043.101417:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513011584:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763043.101452:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513011712:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763043.101481:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763043.101496:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763043/real 1789763043] req@ffff8dc00514b480 x1876699513011712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763043.101510:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b480 x1876699513011712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763043.101521:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763043.101528:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763043.101534:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513011712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763043.101547:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513011840:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763043.101583:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763043.101605:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763043/real 1789763043] req@ffff8dc005148000 x1876699513011840/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763043.101629:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148000 x1876699513011840/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763043.101641:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763043.101645:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763043.101649:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513011840:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763043.101658:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513011968:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763043.101702:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763043.101717:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763043/real 1789763043] req@ffff8dc005148380 x1876699513011968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763043.101731:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148380 x1876699513011968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763043.101740:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763043.101744:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763043.101748:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513011968:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763044.125033:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763045.149049:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763046.173054:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763047.197180:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763048.221071:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763048.221087:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763048.221094:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763048.221113:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3022 00000100:00080000:1.0:1789763048.221130:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763048.221144:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789763048.221151:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763048.221236:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699513012096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763048.221281:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763048.221287:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763048.221290:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763048.221296:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3022 00000100:00080000:1.0:1789763048.221301:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763048.221309:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763048.221326:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763048.221351:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699513012224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763048.221364:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513012096:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789763048.221377:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763048.221384:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763048.221388:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763048.221394:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3022 00000100:00080000:1.0:1789763048.221400:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763048.221405:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763048.221416:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763048.221440:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699513012352/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763048.221453:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763048.221459:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763048.221463:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763048.221470:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3022 00000100:00080000:1.0:1789763048.221475:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789763048.221483:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789763048.221489:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763048.221499:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763048.221517:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699513012480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763048.221566:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763048/real 1789763048] req@ffff8dc005c97800 x1876699513012096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763048.221592:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699513012096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763048.221608:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763048.221613:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763048.221619:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513012096:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763048.221653:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513012224:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763048.221680:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763048.221696:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763048/real 1789763048] req@ffff8dc005c96a00 x1876699513012224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763048.221711:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699513012224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763048.221722:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763048.221726:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763048.221730:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513012224:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763048.221737:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513012352:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789763048.221758:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763048.221771:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763048/real 1789763048] req@ffff8dc005c94700 x1876699513012352/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763048.221785:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699513012352/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763048.221795:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763048.221799:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763048.221802:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513012352:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763048.221809:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513012480:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763048.221848:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763048.221862:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763048/real 1789763048] req@ffff8dc005c95f80 x1876699513012480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763048.221876:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699513012480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763048.221886:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763048.221890:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763048.221894:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513012480:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763049.245209:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763050.269138:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763051.293241:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763052.317269:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763053.341074:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763053.341087:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763053.341093:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763053.341106:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3027 00000100:00080000:2.0:1789763053.341119:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763053.341128:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789763053.341201:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763053.341221:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699513012608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763053.341264:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763053.341270:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763053.341273:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763053.341278:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3027 00000100:00080000:2.0:1789763053.341283:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763053.341289:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763053.341302:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763053.341320:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699513012736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763053.341335:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763053.341339:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763053.341342:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763053.341347:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3027 00000100:00080000:2.0:1789763053.341350:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763053.341354:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763053.341360:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763053.341379:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513012608:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763053.341379:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699513012864/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763053.341408:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763053.341412:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763053.341414:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763053.341422:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3027 00000100:00080000:2.0:1789763053.341426:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763053.341430:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763053.341438:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763053.341453:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699513012992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763053.341484:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763053.341540:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763053/real 1789763053] req@ffff8dc00368d880 x1876699513012608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763053.341564:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699513012608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763053.341578:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763053.341584:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763053.341590:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513012608:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763053.341600:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513012736:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763053.341625:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763053.341639:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763053/real 1789763053] req@ffff8dc00368fb80 x1876699513012736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763053.341655:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699513012736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763053.341665:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763053.341669:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763053.341673:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513012736:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763053.341704:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513012864:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763053.341725:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763053.341737:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763053/real 1789763053] req@ffff8dc00368f480 x1876699513012864/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763053.341770:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699513012864/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763053.341786:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763053.341792:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763053.341799:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513012864:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763053.341810:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513012992:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763053.341850:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763053.341873:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763053/real 1789763053] req@ffff8dc00368f100 x1876699513012992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763053.341890:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699513012992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763053.341933:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763053.341939:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763053.341943:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513012992:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763054.365117:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763055.389183:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763056.413151:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763057.437175:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763058.461024:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763058.461041:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763058.461048:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763058.461067:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3033 00000100:00080000:2.0:1789763058.461082:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763058.461094:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789763058.461175:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763058.461186:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699513013120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763058.461227:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763058.461232:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763058.461235:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763058.461241:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3033 00000100:00080000:2.0:1789763058.461248:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763058.461254:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763058.461269:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763058.461295:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699513013248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763058.461297:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513013120:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763058.461323:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763058.461330:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763058.461333:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763058.461339:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3033 00000100:00080000:2.0:1789763058.461344:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763058.461348:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763058.461355:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763058.461373:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699513013376/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763058.461387:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763058.461393:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763058.461396:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763058.461401:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3033 00000100:00080000:2.0:1789763058.461405:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763058.461410:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1789763058.461420:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:2.0:1789763058.461422:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763058.461442:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699513013504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763058.461489:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763058/real 1789763058] req@ffff8dc00368ce00 x1876699513013120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763058.461518:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699513013120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763058.461549:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763058.461557:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763058.461566:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513013120:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763058.461609:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513013248:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763058.461653:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763058.461678:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763058/real 1789763058] req@ffff8dc00368c000 x1876699513013248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763058.461701:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699513013248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763058.461716:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763058.461721:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763058.461727:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513013248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763058.461740:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513013376:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763058.461801:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763058.461824:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763058/real 1789763058] req@ffff8dc00368ea00 x1876699513013376/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763058.461850:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699513013376/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763058.461862:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763058.461867:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763058.461871:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513013376:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763058.461880:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513013504:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763058.461951:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763058.461968:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763058/real 1789763058] req@ffff8dc00368ca80 x1876699513013504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763058.461984:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699513013504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763058.461994:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763058.461998:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763058.462002:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513013504:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763059.485181:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763060.509131:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763061.533140:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763062.557212:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763063.581136:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763063.581170:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763063.581198:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763063.581205:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763063.581222:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3038 00000100:00080000:3.0:1789763063.581238:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763063.581250:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763063.581312:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699513013632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763063.581358:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763063.581364:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763063.581366:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763063.581374:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3038 00000100:00080000:3.0:1789763063.581381:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763063.581387:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763063.581405:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763063.581430:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513013632:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763063.581431:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699513013760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763063.581475:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763063.581479:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763063.581481:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763063.581487:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3038 00000100:00080000:3.0:1789763063.581492:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763063.581497:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763063.581507:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763063.581525:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699513013888/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763063.581548:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763063.581554:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763063.581558:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789763063.581566:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763063.581621:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763063/real 1789763063] req@ffff8dc0107ad180 x1876699513013632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763063.581645:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699513013632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763063.581659:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763063.581665:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763063.581671:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513013632:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763063.581689:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3038 00000100:00080000:3.0:1789763063.581697:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763063.581703:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763063.581710:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513013760:192.168.203.156@tcp:38:kworker.0 00000008:00100000:3.0:1789763063.581711:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763063.581763:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763063.581789:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763063/real 1789763063] req@ffff8dc0107af480 x1876699513013760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763063.581815:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699513013760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763063.581829:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763063.581833:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763063.581837:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513013760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763063.581845:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513013888:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763063.581878:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763063.581895:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763063/real 1789763063] req@ffff8dc0107adf80 x1876699513013888/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763063.581909:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699513013888/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763063.581985:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789763063.581990:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699513014016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763063.581992:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763063.582005:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513013888:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763063.582185:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513014016:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763063.582221:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763063.582237:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763063/real 1789763063] req@ffff8dc0107ac380 x1876699513014016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763063.582251:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699513014016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763063.582261:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763063.582264:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763063.582268:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513014016:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763064.605219:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763065.629124:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763066.653119:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763067.677166:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763068.701019:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763068.701027:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763068.701031:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763068.701055:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3043 00000100:00080000:0.0:1789763068.701065:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763068.701074:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763068.701119:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149c00 x1876699513014144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763068.701146:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763068.701149:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763068.701151:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763068.701154:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3043 00000100:00080000:0.0:1789763068.701156:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763068.701159:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763068.701166:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763068.701176:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149880 x1876699513014272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763068.701184:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763068.701187:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763068.701188:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763068.701191:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3043 00000100:00080000:0.0:1789763068.701193:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763068.701196:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763068.701200:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763068.701204:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513014144:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763068.701211:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b800 x1876699513014400/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763068.701219:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763068.701222:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763068.701223:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763068.701226:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3043 00000100:00080000:0.0:1789763068.701228:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763068.701230:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763068.701233:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763068.701243:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149180 x1876699513014528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763068.701268:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00010000:00010000:3.0:1789763068.701307:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763068.701313:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763068/real 1789763068] req@ffff8dc005149c00 x1876699513014144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763068.701330:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149c00 x1876699513014144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763068.701340:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763068.701344:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763068.701350:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513014144:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763068.701357:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513014272:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763068.701371:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763068.701379:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763068/real 1789763068] req@ffff8dc005149880 x1876699513014272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763068.701412:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149880 x1876699513014272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763068.701423:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763068.701427:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763068.701431:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513014272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763068.701461:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513014400:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763068.701485:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763068.701497:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763068/real 1789763068] req@ffff8dc00514b800 x1876699513014400/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763068.701512:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b800 x1876699513014400/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763068.701525:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763068.701530:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763068.701537:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513014400:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763068.701549:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513014528:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763068.701586:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763068.701608:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763068/real 1789763068] req@ffff8dc005149180 x1876699513014528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763068.701633:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149180 x1876699513014528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763068.701661:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763068.701665:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763068.701669:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513014528:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763069.725150:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763070.749177:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763071.773156:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763072.797047:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763073.821180:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763073.821196:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763073.821214:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763073.821233:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3048 00000100:00080000:3.0:1789763073.821248:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763073.821259:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789763073.821263:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763073.821337:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699513014656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763073.821379:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763073.821385:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763073.821388:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763073.821405:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3048 00000100:00080000:3.0:1789763073.821411:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763073.821417:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763073.821432:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763073.821452:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513014656:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763073.821455:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699513014784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763073.821487:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763073.821493:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763073.821495:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763073.821501:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3048 00000100:00080000:3.0:1789763073.821505:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763073.821509:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763073.821516:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763073.821534:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699513014912/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763073.821549:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763073.821554:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763073.821556:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763073.821561:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3048 00000100:00080000:3.0:1789763073.821568:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763073.821575:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763073.821582:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763073.821602:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763073.821615:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699513015040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763073.821678:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763073/real 1789763073] req@ffff8dc0107af800 x1876699513014656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763073.821720:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699513014656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763073.821737:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763073.821742:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763073.821748:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513014656:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763073.821782:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513014784:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763073.821819:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763073.821835:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763073/real 1789763073] req@ffff8dc0107ae680 x1876699513014784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763073.821850:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699513014784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763073.821861:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763073.821866:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763073.821870:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513014784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763073.821879:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513014912:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763073.821946:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763073.821962:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763073/real 1789763073] req@ffff8dc0107ace00 x1876699513014912/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763073.821977:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699513014912/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763073.821988:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763073.821992:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763073.821996:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513014912:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763073.822050:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513015040:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763073.822088:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763073.822111:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763073/real 1789763073] req@ffff8dc0107af100 x1876699513015040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763073.822133:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699513015040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763073.822144:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763073.822148:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763073.822152:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513015040:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763074.845067:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763075.869039:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763076.893130:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763077.917228:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763078.941128:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763078.941147:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:1.0:1789763078.941151:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763078.941154:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763078.941175:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3053 00000100:00080000:2.0:1789763078.941195:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763078.941209:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763078.941263:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699513015168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763078.941301:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763078.941306:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763078.941309:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763078.941314:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3053 00000100:00080000:2.0:1789763078.941319:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763078.941324:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763078.941338:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763078.941357:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699513015296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763078.941369:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513015168:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763078.941380:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763078.941391:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763078.941393:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763078.941400:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3053 00000100:00080000:2.0:1789763078.941404:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763078.941409:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763078.941415:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763078.941433:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699513015424/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763078.941447:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763078.941452:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763078.941456:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763078.941461:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3053 00000100:00080000:2.0:1789763078.941466:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763078.941472:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763078.941478:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763078.941491:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699513015552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763078.941554:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513015296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763078.941626:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513015424:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763078.941669:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513015552:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763079.965261:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763080.989275:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763082.013397:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763082.013429:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:2.0:1789763082.013487:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763082.013508:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789763082.013562:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789763082.013595:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789763082.013615:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:1.0:1789763082.013668:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763078/real 1789763082] req@ffff8dc00368ed80 x1876699513015296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763082.013720:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699513015296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763082.013743:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763082.013752:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763082.013760:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513015296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763082.013776:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763078/real 1789763082] req@ffff8dc00368e300 x1876699513015424/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763082.013815:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699513015424/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763082.013834:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763082.013841:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763082.013848:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513015424:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763082.013861:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763078/real 1789763082] req@ffff8dc00368f800 x1876699513015552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763082.014008:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699513015552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763082.014028:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763082.014035:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763082.014043:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513015552:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763082.014056:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763078/real 1789763082] req@ffff8dc00368df80 x1876699513015168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763082.014083:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699513015168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763082.014100:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763082.014106:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763082.014114:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513015168:192.168.203.156@tcp:250:kworker.0 00010000:00010000:2.0:1789763083.037241:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763084.061120:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763084.061137:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763084.061145:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763084.061177:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3058 00000100:00080000:2.0:1789763084.061195:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763084.061209:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789763084.061234:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763084.061269:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699513015680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763084.061314:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763084.061319:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763084.061323:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763084.061328:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3058 00000100:00080000:2.0:1789763084.061334:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763084.061339:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763084.061364:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763084.061386:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699513015808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763084.061414:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513015680:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763084.061420:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763084.061437:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763084.061441:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763084.061463:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3058 00000100:00080000:2.0:1789763084.061470:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763084.061478:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763084.061490:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763084.061523:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699513015936/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763084.061547:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763084.061554:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763084.061559:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763084.061569:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3058 00000800:00000100:1.0:1789763084.061579:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789763084.061585:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763084.061600:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763084.061614:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763084.061645:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699513016064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763084.061660:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763084/real 1789763084] req@ffff8dc00368dc00 x1876699513015680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763084.061725:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699513015680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763084.061744:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763084.061752:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763084.061758:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513015680:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763084.061797:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513015808:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763084.061829:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763084.061847:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763084/real 1789763084] req@ffff8dc00368e680 x1876699513015808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763084.061861:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699513015808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763084.061870:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763084.061874:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763084.061877:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513015808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763084.061885:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513015936:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763084.061962:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763084.061983:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763084/real 1789763084] req@ffff8dc00368d180 x1876699513015936/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763084.061997:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699513015936/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763084.062007:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763084.062011:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763084.062015:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513015936:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763084.062023:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513016064:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763084.062047:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763084.062059:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763084/real 1789763084] req@ffff8dc00368c700 x1876699513016064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763084.062074:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699513016064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763084.062083:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763084.062087:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763084.062091:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513016064:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763085.085257:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763086.109176:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763087.133142:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763088.157138:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763089.181049:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763089.181062:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763089.181068:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763089.181081:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3063 00000100:00080000:2.0:1789763089.181095:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763089.181104:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789763089.181133:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763089.181160:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699513016192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763089.181199:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763089.181204:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763089.181207:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763089.181212:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3063 00000100:00080000:2.0:1789763089.181216:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763089.181220:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763089.181231:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763089.181248:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699513016320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763089.181261:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763089.181266:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763089.181269:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763089.181274:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3063 00000100:00080000:2.0:1789763089.181280:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763089.181284:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763089.181291:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763089.181302:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513016192:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763089.181307:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699513016448/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763089.181327:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763089.181330:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763089.181332:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763089.181338:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3063 00000100:00080000:2.0:1789763089.181342:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763089.181346:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763089.181351:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763089.181367:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699513016576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763089.181414:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763089.181476:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763089/real 1789763089] req@ffff8dc00368c700 x1876699513016192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763089.181501:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699513016192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763089.181515:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763089.181521:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763089.181526:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513016192:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763089.181536:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513016320:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763089.181566:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763089.181581:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763089/real 1789763089] req@ffff8dc00368d180 x1876699513016320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763089.181596:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699513016320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763089.181606:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763089.181611:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763089.181615:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513016320:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763089.181644:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513016448:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763089.181666:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763089.181678:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763089/real 1789763089] req@ffff8dc00368e680 x1876699513016448/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763089.181692:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699513016448/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763089.181703:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763089.181706:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763089.181710:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513016448:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763089.181717:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513016576:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763089.181734:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763089.181745:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763089/real 1789763089] req@ffff8dc00368dc00 x1876699513016576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763089.181759:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699513016576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763089.181768:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763089.181771:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763089.181776:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513016576:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763090.205140:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763091.229103:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763092.253129:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763093.277127:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763094.301127:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1789763094.301137:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763094.301141:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763094.301151:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763094.301166:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3068 00000100:00080000:1.0:1789763094.301184:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763094.301199:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763094.301253:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699513016704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763094.301302:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763094.301307:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763094.301310:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763094.301315:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3068 00000100:00080000:1.0:1789763094.301320:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763094.301326:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763094.301341:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763094.301357:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699513016832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763094.301374:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763094.301382:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789763094.301384:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513016704:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789763094.301387:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763094.301412:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3068 00000100:00080000:1.0:1789763094.301417:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763094.301421:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763094.301431:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763094.301459:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699513016960/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763094.301479:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763094.301486:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763094.301491:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763094.301501:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3068 00000100:00080000:1.0:1789763094.301510:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763094.301520:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789763094.301534:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1789763094.301543:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763094.301579:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699513017088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763094.301613:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763094/real 1789763094] req@ffff8dc005c94380 x1876699513016704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763094.301644:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699513016704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763094.301661:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763094.301668:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763094.301674:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513016704:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763094.301711:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513016832:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763094.301739:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763094.301755:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763094/real 1789763094] req@ffff8dc005c97b80 x1876699513016832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763094.301771:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699513016832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763094.301782:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763094.301786:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763094.301790:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513016832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763094.301797:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513016960:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763094.301819:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763094.301831:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763094/real 1789763094] req@ffff8dc005c96300 x1876699513016960/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763094.301846:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699513016960/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763094.301891:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763094.301896:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763094.301934:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513016960:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763094.301942:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513017088:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763094.301969:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763094.301984:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763094/real 1789763094] req@ffff8dc005c95500 x1876699513017088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763094.301999:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699513017088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763094.302009:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763094.302013:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763094.302017:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513017088:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763095.325142:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763096.349138:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763097.373137:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763098.397120:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763099.421093:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763099.421105:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763099.421111:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763099.421124:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3074 00010000:00010000:1.0:1789763099.421129:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763099.421138:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763099.421150:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763099.421208:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699513017216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763099.421238:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763099.421243:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763099.421246:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763099.421253:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3074 00000100:00080000:2.0:1789763099.421261:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763099.421267:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763099.421279:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763099.421292:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699513017344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763099.421302:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763099.421306:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763099.421307:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763099.421311:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3074 00000100:00080000:2.0:1789763099.421313:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763099.421316:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763099.421321:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763099.421331:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699513017472/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763099.421340:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763099.421342:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763099.421343:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763099.421346:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3074 00000100:00080000:2.0:1789763099.421350:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763099.421353:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763099.421358:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763099.421366:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699513017600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763099.421494:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513017216:192.168.203.156@tcp:250:kworker.0 00000800:00000100:0.0:1789763099.421607:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763099.421681:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763099/real 1789763099] req@ffff8dc00368df80 x1876699513017216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763099.421712:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699513017216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763099.421738:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763099.421745:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763099.421752:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513017216:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763099.421764:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513017344:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763099.421798:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763099.421836:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763099/real 1789763099] req@ffff8dc00368f800 x1876699513017344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763099.421853:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699513017344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763099.421864:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763099.421868:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763099.421873:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513017344:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763099.421882:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513017472:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763099.421944:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763099.421972:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763099/real 1789763099] req@ffff8dc00368e300 x1876699513017472/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763099.421997:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699513017472/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763099.422011:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763099.422017:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763099.422023:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513017472:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763099.422033:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513017600:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763099.422070:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763099.422089:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763099/real 1789763099] req@ffff8dc00368ed80 x1876699513017600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763099.422105:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699513017600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763099.422113:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763099.422115:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763099.422118:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513017600:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763100.445118:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763101.469143:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763102.493142:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763103.517104:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763104.541093:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763104.541116:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763104.541124:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789763104.541137:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763104.541149:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3079 00000100:00080000:2.0:1789763104.541171:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763104.541185:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763104.541253:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699513017728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763104.541295:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763104.541301:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763104.541303:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763104.541309:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3079 00000100:00080000:2.0:1789763104.541314:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763104.541319:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763104.541334:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763104.541355:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699513017856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763104.541373:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763104.541380:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763104.541382:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763104.541390:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3079 00000100:00080000:2.0:1789763104.541395:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763104.541407:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513017728:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763104.541407:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763104.541417:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763104.541445:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699513017984/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763104.541461:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763104.541466:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763104.541469:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763104.541476:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3079 00000100:00080000:2.0:1789763104.541479:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763104.541486:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763104.541491:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763104.541505:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699513018112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763104.541533:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763104.541593:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763104/real 1789763104] req@ffff8dc00368ca80 x1876699513017728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763104.541618:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699513017728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763104.541633:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763104.541639:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763104.541644:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513017728:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763104.541677:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513017856:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763104.541719:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763104.541745:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763104/real 1789763104] req@ffff8dc00368ea00 x1876699513017856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763104.541795:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699513017856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763104.541834:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763104.541842:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763104.541850:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513017856:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763104.541864:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513017984:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763104.541953:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763104.541973:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763104/real 1789763104] req@ffff8dc00368c000 x1876699513017984/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763104.541997:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699513017984/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763104.542014:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763104.542021:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763104.542027:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513017984:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763104.542041:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513018112:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763104.542085:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763104.542104:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763104/real 1789763104] req@ffff8dc00368ce00 x1876699513018112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763104.542119:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699513018112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763104.542130:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763104.542134:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763104.542139:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513018112:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763105.565219:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763106.589049:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763107.613050:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763108.637201:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763109.661080:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763109.661101:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763109.661106:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763109.661120:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3084 00010000:00010000:1.0:1789763109.661124:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763109.661131:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763109.661141:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763109.661190:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699513018240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763109.661220:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763109.661223:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763109.661224:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763109.661228:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3084 00000100:00080000:3.0:1789763109.661231:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763109.661234:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763109.661246:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763109.661264:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699513018368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763109.661279:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763109.661284:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763109.661287:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763109.661298:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3084 00000100:00100000:0.0:1789763109.661300:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513018240:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763109.661304:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763109.661317:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763109.661332:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763109.661350:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699513018496/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763109.661361:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763109.661364:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763109.661366:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763109.661369:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3084 00000100:00080000:3.0:1789763109.661372:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763109.661378:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763109.661383:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763109.661400:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699513018624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763109.661425:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763109.661484:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763109/real 1789763109] req@ffff8dc0107aea00 x1876699513018240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763109.661536:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699513018240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763109.661551:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763109.661557:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763109.661562:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513018240:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763109.661598:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513018368:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763109.661627:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763109.661643:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763109/real 1789763109] req@ffff8dc0107ad880 x1876699513018368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763109.661659:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699513018368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763109.661670:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763109.661675:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763109.661679:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513018368:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763109.661687:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513018496:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763109.661712:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763109.661724:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763109/real 1789763109] req@ffff8dc0107af100 x1876699513018496/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763109.661770:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699513018496/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763109.661782:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763109.661796:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763109.661801:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513018496:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763109.661809:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513018624:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763109.661836:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763109.661850:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763109/real 1789763109] req@ffff8dc0107ace00 x1876699513018624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763109.661866:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699513018624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763109.661883:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763109.661890:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763109.661896:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513018624:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763110.685073:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763111.709150:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763112.733192:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763113.757082:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763114.781119:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1789763114.781140:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763114.781141:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763114.781152:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763114.781171:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3089 00000100:00080000:1.0:1789763114.781188:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763114.781200:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763114.781267:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699513018752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763114.781315:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763114.781321:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763114.781323:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763114.781329:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3089 00000100:00080000:1.0:1789763114.781333:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763114.781349:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763114.781367:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763114.781391:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699513018880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763114.781399:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513018752:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789763114.781440:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763114.781446:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763114.781450:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763114.781461:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3089 00000100:00080000:1.0:1789763114.781471:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763114.781481:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763114.781493:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789763114.781520:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763114.781525:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699513019008/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763114.781556:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763114.781564:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763114.781569:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763114.781579:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3089 00000100:00080000:1.0:1789763114.781587:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763114.781597:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763114.781601:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763114/real 1789763114] req@ffff8dc005c94e00 x1876699513018752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1789763114.781607:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763114.781635:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699513019136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763114.781656:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699513018752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763114.781674:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763114.781680:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763114.781686:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513018752:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763114.781752:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513018880:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763114.781798:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763114.781818:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763114/real 1789763114] req@ffff8dc005c97480 x1876699513018880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763114.781834:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699513018880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763114.781846:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763114.781851:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763114.781855:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513018880:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763114.781863:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513019008:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763114.781886:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763114.781931:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763114/real 1789763114] req@ffff8dc005c96680 x1876699513019008/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763114.781949:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699513019008/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763114.781969:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763114.781973:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763114.781978:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513019008:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763114.781986:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513019136:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763114.782021:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763114.782043:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763114/real 1789763114] req@ffff8dc005c95880 x1876699513019136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763139 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763114.782069:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699513019136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763139 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763114.782087:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763114.782094:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763114.782101:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513019136:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763115.805183:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763116.829100:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763117.853153:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763118.877189:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763119.901084:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763119.901106:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763119.901124:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789763119.901127:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763119.901140:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3094 00000100:00080000:3.0:1789763119.901156:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763119.901169:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763119.901245:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699513019264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763119.901289:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763119.901297:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763119.901300:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763119.901308:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3094 00000100:00080000:3.0:1789763119.901315:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763119.901335:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763119.901353:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763119.901376:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513019264:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763119.901384:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699513019392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763119.901413:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763119.901417:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763119.901420:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763119.901424:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3094 00000100:00080000:3.0:1789763119.901429:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763119.901434:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763119.901443:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763119.901458:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699513019520/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763119.901471:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763119.901476:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763119.901478:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763119.901492:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3094 00000100:00080000:3.0:1789763119.901496:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763119.901499:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763119.901505:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763119.901519:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699513019648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763119.901528:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763119.901606:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763119/real 1789763119] req@ffff8dc0107ae680 x1876699513019264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763119.901642:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699513019264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763119.901657:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763119.901686:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763119.901692:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513019264:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763119.901736:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513019392:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763119.901767:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763119.901783:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763119/real 1789763119] req@ffff8dc0107af800 x1876699513019392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763119.901798:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699513019392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763119.901808:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763119.901812:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763119.901816:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513019392:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763119.901822:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513019520:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763119.901842:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763119.901853:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763119/real 1789763119] req@ffff8dc0107ac380 x1876699513019520/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763119.901866:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699513019520/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763119.901876:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763119.901880:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763119.901883:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513019520:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763119.901889:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513019648:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763119.901955:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763119.901970:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763119/real 1789763119] req@ffff8dc0107adf80 x1876699513019648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763119.901984:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699513019648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763119.901993:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763119.901997:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763119.902001:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513019648:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763120.925165:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763121.949126:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763122.973120:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763123.997138:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763125.021080:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763125.021095:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763125.021101:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763125.021118:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3099 00000100:00080000:3.0:1789763125.021134:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763125.021163:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763125.021201:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763125.021220:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699513019776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763125.021252:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763125.021256:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763125.021259:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763125.021263:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3099 00000100:00080000:3.0:1789763125.021269:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763125.021274:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763125.021286:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763125.021303:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699513019904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763125.021307:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513019776:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763125.021324:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763125.021329:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763125.021331:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763125.021336:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3099 00000100:00080000:3.0:1789763125.021339:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763125.021343:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763125.021349:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763125.021363:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699513020032/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763125.021374:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763125.021377:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763125.021380:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763125.021386:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3099 00000100:00080000:3.0:1789763125.021390:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763125.021394:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789763125.021403:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789763125.021405:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763125.021421:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699513020160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763125.021461:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763125/real 1789763125] req@ffff8dc0107af480 x1876699513019776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763125.021484:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699513019776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763125.021496:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763125.021501:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763125.021506:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513019776:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763125.021532:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513019904:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763125.021573:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763125.021587:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763125/real 1789763125] req@ffff8dc0107ad180 x1876699513019904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763125.021650:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699513019904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763125.021660:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763125.021664:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763125.021667:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513019904:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763125.021674:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513020032:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763125.021697:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763125.021709:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763125/real 1789763125] req@ffff8dc0107ac700 x1876699513020032/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763125.021721:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699513020032/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763125.021728:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763125.021732:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763125.021735:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513020032:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763125.021740:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513020160:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763125.021754:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763125.021763:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763125/real 1789763125] req@ffff8dc0107aed80 x1876699513020160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763125.021774:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699513020160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763125.021787:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763125.021790:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763125.021793:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513020160:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763126.045098:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763127.069171:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763128.093354:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763129.117039:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763130.141089:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:1.0:1789763130.141096:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763130.141109:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763130.141116:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763130.141154:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3104 00000100:00080000:3.0:1789763130.141174:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763130.141188:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763130.141282:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699513020288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763130.141340:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763130.141348:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763130.141353:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763130.141364:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3104 00000100:00080000:3.0:1789763130.141388:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763130.141400:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513020288:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763130.141407:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763130.141448:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763130.141477:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699513020416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763130.141495:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763130.141502:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763130.141506:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763130.141513:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3104 00000800:00000100:0.0:1789763130.141522:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763130.141524:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763130.141535:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763130.141547:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763130.141577:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699513020544/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763130.141603:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763130/real 1789763130] req@ffff8dc0107ad500 x1876699513020288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763130.141660:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763130.141669:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699513020288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763130.141671:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763130.141680:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763130.141703:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3104 00000100:00080000:0.0:1789763130.141707:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763130.141715:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763130.141716:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763130.141733:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513020288:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763130.141740:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763130.141756:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763130.141772:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513020416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789763130.141784:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699513020672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763130.141803:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763130.141829:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763130/real 1789763130] req@ffff8dc0107ac000 x1876699513020416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763130.141855:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699513020416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763130.141866:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763130.141871:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763130.141875:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513020416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763130.141883:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513020544:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763130.141928:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763130.141941:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763130/real 1789763130] req@ffff8dc0107ae300 x1876699513020544/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763130.141954:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699513020544/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763130.141962:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763130.141966:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763130.141969:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513020544:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763130.141984:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513020672:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763130.142015:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763130.142027:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763130/real 1789763130] req@ffff8dc0107aca80 x1876699513020672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763130.142038:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699513020672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763130.142046:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763130.142049:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763130.142052:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513020672:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763131.165187:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763132.189166:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763133.213160:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763134.237131:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763135.261010:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763135.261050:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763135.261057:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763135.261060:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763135.261068:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3109 00000100:00080000:1.0:1789763135.261077:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763135.261083:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763135.261118:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96d80 x1876699513020800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763135.261140:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763135.261144:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763135.261147:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763135.261152:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3109 00000100:00080000:1.0:1789763135.261155:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763135.261159:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763135.261168:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763135.261179:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513020800:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763135.261185:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699513020928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763135.261201:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763135.261203:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763135.261205:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763135.261208:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3109 00000100:00080000:1.0:1789763135.261215:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763135.261218:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763135.261224:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763135.261235:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699513021056/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763135.261243:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763135.261246:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763135.261248:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763135.261254:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3109 00000100:00080000:1.0:1789763135.261257:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763135.261261:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763135.261264:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763135.261272:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699513021184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763135.261306:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763135.261633:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763135/real 1789763135] req@ffff8dc005c96d80 x1876699513020800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763135.261707:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96d80 x1876699513020800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763135.261723:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763135.261728:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763135.261732:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513020800:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763135.261757:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513020928:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763135.261780:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763135.261791:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763135/real 1789763135] req@ffff8dc005c95880 x1876699513020928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763135.261800:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699513020928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763135.261806:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763135.261809:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763135.261811:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513020928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789763135.261815:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513021056:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789763135.261829:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763135.261835:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763135/real 1789763135] req@ffff8dc005c96680 x1876699513021056/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763135.261843:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699513021056/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763135.261848:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763135.261850:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763135.261852:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513021056:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789763135.261856:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513021184:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763135.261870:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763135.261880:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763135/real 1789763135] req@ffff8dc005c97480 x1876699513021184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763135.261952:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699513021184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763135.261967:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763135.261972:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763135.261976:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513021184:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763136.285211:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763137.309140:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763138.333229:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763139.357136:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763140.381093:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763140.381114:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763140.381123:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1789763140.381141:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763140.381145:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3114 00000100:00080000:1.0:1789763140.381168:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763140.381183:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763140.381243:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699513021312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763140.381295:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763140.381302:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763140.381307:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763140.381315:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3114 00000100:00080000:1.0:1789763140.381327:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763140.381334:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763140.381348:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763140.381370:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513021312:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763140.381374:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699513021440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763140.381413:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763140.381417:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763140.381420:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763140.381428:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3114 00000100:00080000:1.0:1789763140.381432:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763140.381437:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763140.381444:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763140.381459:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699513021568/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763140.381473:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763140.381478:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763140.381480:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763140.381485:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3114 00000100:00080000:1.0:1789763140.381489:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763140.381494:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763140.381500:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763140.381542:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763140.381546:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699513021696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763140.381618:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763140/real 1789763140] req@ffff8dc005c94e00 x1876699513021312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763140.381648:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699513021312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763140.381666:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763140.381673:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763140.381681:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513021312:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763140.381721:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513021440:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763140.381753:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763140.381770:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763140/real 1789763140] req@ffff8dc005c95500 x1876699513021440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763140.381788:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699513021440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763140.381816:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763140.381823:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763140.381831:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513021440:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789763140.381844:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513021568:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789763140.381887:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763140.381947:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763140/real 1789763140] req@ffff8dc005c96300 x1876699513021568/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763140.381965:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699513021568/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763140.381976:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763140.381981:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763140.381985:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513021568:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789763140.381996:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513021696:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763140.382038:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763140.382063:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763140/real 1789763140] req@ffff8dc005c97b80 x1876699513021696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763140.382089:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699513021696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763140.382104:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763140.382109:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763140.382114:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513021696:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763141.405166:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763142.429220:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763143.453149:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763144.477130:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763145.501099:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763145.501110:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763145.501115:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763145.501127:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3120 00000100:00080000:3.0:1789763145.501139:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763145.501147:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789763145.501201:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763145.501229:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699513021824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763145.501275:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763145.501281:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763145.501285:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763145.501291:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3120 00000100:00080000:3.0:1789763145.501298:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763145.501305:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763145.501320:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763145.501343:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699513021952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763145.501360:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513021824:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763145.501364:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763145.501388:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763145.501392:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763145.501401:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3120 00000100:00080000:3.0:1789763145.501409:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763145.501419:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763145.501429:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763145.501455:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699513022080/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763145.501490:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763145.501496:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763145.501498:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763145.501521:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3120 00000100:00080000:3.0:1789763145.501529:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763145.501534:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763145.501542:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763145.501563:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699513022208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763145.501641:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513021952:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763145.501703:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513022080:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763145.501737:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513022208:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763146.525168:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763147.549240:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763148.573224:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789763148.573492:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1789763148.573570:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789763148.573599:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789763148.573683:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789763148.573702:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789763148.573720:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:0.0:1789763148.573776:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763145/real 1789763148] req@ffff8dc0107adc00 x1876699513021952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763148.573829:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699513021952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763148.573854:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763148.573863:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763148.573872:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513021952:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763148.573888:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763145/real 1789763148] req@ffff8dc0107aca80 x1876699513022080/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763148.573964:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699513022080/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763148.573985:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763148.573994:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763148.574002:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513022080:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763148.574031:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763145/real 1789763148] req@ffff8dc0107ae300 x1876699513022208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763148.574059:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699513022208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763148.574102:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763148.574113:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763148.574125:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513022208:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763148.574145:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763145/real 1789763148] req@ffff8dc0107afb80 x1876699513021824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763148.574174:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699513021824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763148.574190:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763148.574197:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763148.574204:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513021824:192.168.203.156@tcp:250:kworker.0 00010000:00010000:2.0:1789763149.597194:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763150.621042:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763150.621059:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763150.621066:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763150.621087:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3125 00000100:00080000:3.0:1789763150.621111:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763150.621127:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763150.621198:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763150.621209:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699513022336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763150.621253:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763150.621259:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763150.621261:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763150.621267:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3125 00000100:00080000:3.0:1789763150.621273:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763150.621278:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763150.621294:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763150.621318:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699513022464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763150.621341:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763150.621348:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513022336:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763150.621350:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763150.621359:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763150.621377:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3125 00000100:00080000:3.0:1789763150.621384:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763150.621391:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763150.621402:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763150.621461:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699513022592/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763150.621484:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763150.621492:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763150.621498:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763150.621505:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3125 00000100:00080000:3.0:1789763150.621514:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789763150.621527:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763150.621528:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763150.621544:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763150.621566:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699513022720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763150.621621:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763150/real 1789763150] req@ffff8dc0107ac000 x1876699513022336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763150.621662:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699513022336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763150.621685:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763150.621694:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763150.621704:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513022336:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763150.621753:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513022464:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763150.621812:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763150.621842:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763150/real 1789763150] req@ffff8dc0107ad500 x1876699513022464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763150.621871:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699513022464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763150.621888:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763150.621895:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763150.621947:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513022464:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763150.621963:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513022592:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763150.622014:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763150.622040:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763150/real 1789763150] req@ffff8dc0107aed80 x1876699513022592/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763150.622067:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699513022592/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763150.622084:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763150.622091:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763150.622098:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513022592:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763150.622112:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513022720:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763150.622156:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763150.622183:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763150/real 1789763150] req@ffff8dc0107ac700 x1876699513022720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763150.622225:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699513022720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763150.622242:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763150.622248:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763150.622255:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513022720:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763151.645084:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763152.669229:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763153.693255:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763154.717260:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763155.740994:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763155.741005:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763155.741010:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763155.741023:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3130 00000100:00080000:3.0:1789763155.741037:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763155.741045:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763155.741099:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699513022848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763155.741142:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763155.741148:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763155.741152:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763155.741158:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3130 00000100:00080000:3.0:1789763155.741174:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763155.741179:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763155.741190:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513022848:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789763155.741193:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763155.741222:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699513022976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763155.741250:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:1.0:1789763155.741254:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763155.741256:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763155.741264:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763155.741271:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3130 00000100:00080000:3.0:1789763155.741277:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763155.741282:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763155.741290:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789763155.741306:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763155.741312:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699513023104/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763155.741330:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763155.741351:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763155.741354:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763155.741358:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3130 00000100:00080000:3.0:1789763155.741361:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763155.741366:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763155.741372:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763155.741381:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763155/real 1789763155] req@ffff8dc0107ad180 x1876699513022848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763155.741387:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699513023232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763155.741457:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699513022848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763155.741471:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763155.741475:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763155.741480:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513022848:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763155.741517:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513022976:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763155.741549:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763155.741569:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763155/real 1789763155] req@ffff8dc0107af480 x1876699513022976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763155.741589:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699513022976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763155.741615:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763155.741620:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763155.741625:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513022976:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763155.741632:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513023104:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763155.741667:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763155.741679:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763155/real 1789763155] req@ffff8dc0107adf80 x1876699513023104/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763155.741690:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699513023104/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763155.741698:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763155.741701:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763155.741704:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513023104:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763155.741709:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513023232:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763155.741745:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763155.741757:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763155/real 1789763155] req@ffff8dc0107ac380 x1876699513023232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763180 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763155.741770:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699513023232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763180 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763155.741780:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763155.741783:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763155.741788:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513023232:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763156.765143:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763157.789182:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763158.813179:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763159.837147:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763160.861155:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:1.0:1789763160.861171:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763160.861173:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763160.861185:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763160.861203:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3135 00000100:00080000:2.0:1789763160.861217:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763160.861227:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763160.861279:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699513023360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763160.861315:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763160.861320:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763160.861323:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763160.861329:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3135 00000100:00080000:2.0:1789763160.861333:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763160.861338:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763160.861350:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763160.861393:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699513023488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763160.861410:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513023360:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763160.861416:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763160.861428:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763160.861434:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763160.861441:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3135 00000100:00080000:2.0:1789763160.861446:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763160.861453:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763160.861461:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763160.861479:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699513023616/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763160.861493:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763160.861498:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763160.861501:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763160.861506:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3135 00000100:00080000:2.0:1789763160.861510:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763160.861516:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789763160.861526:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:2.0:1789763160.861528:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763160.861548:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699513023744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763160.861602:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763160/real 1789763160] req@ffff8dc00368f100 x1876699513023360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763160.861632:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699513023360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763160.861647:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763160.861652:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763160.861658:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513023360:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763160.861692:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513023488:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763160.861725:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763160.861742:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763160/real 1789763160] req@ffff8dc00368f480 x1876699513023488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763160.861757:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699513023488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763160.861768:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763160.861774:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763160.861780:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513023488:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763160.861790:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513023616:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763160.861813:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763160.861826:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763160/real 1789763160] req@ffff8dc00368fb80 x1876699513023616/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763160.861841:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699513023616/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763160.861851:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763160.861855:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763160.861859:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513023616:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763160.861865:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513023744:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763160.861884:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763160.861894:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763160/real 1789763160] req@ffff8dc00368d880 x1876699513023744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763160.861943:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699513023744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763160.861954:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763160.861958:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763160.861961:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513023744:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763161.885203:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763162.909151:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763163.933256:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763164.957169:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763165.981082:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763165.981099:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763165.981118:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763165.981136:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3140 00000100:00080000:3.0:1789763165.981154:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763165.981167:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789763165.981204:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763165.981241:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699513023872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763165.981286:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763165.981293:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763165.981295:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763165.981301:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3140 00000100:00080000:3.0:1789763165.981306:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763165.981312:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763165.981360:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763165.981388:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699513024000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763165.981389:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513023872:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763165.981416:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763165.981421:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763165.981423:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763165.981430:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3140 00000100:00080000:3.0:1789763165.981435:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763165.981440:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763165.981448:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763165.981464:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699513024128/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763165.981487:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763165.981492:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763165.981494:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763165.981504:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3140 00000800:00000100:0.0:1789763165.981505:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763165.981511:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763165.981519:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763165.981526:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763165.981541:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699513024256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763165.981586:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763165/real 1789763165] req@ffff8dc0107af800 x1876699513023872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763165.981616:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699513023872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763165.981633:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763165.981639:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763165.981656:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513023872:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763165.981690:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513024000:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763165.981719:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763165.981748:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763165/real 1789763165] req@ffff8dc0107ae680 x1876699513024000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763165.981766:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699513024000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763165.981777:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763165.981781:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763165.981786:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513024000:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763165.981794:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513024128:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763165.981817:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763165.981831:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763165/real 1789763165] req@ffff8dc0107ace00 x1876699513024128/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763165.981845:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699513024128/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763165.981856:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763165.981882:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763165.981887:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513024128:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763165.981895:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513024256:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763165.981967:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763165.981994:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763165/real 1789763165] req@ffff8dc0107af100 x1876699513024256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763165.982020:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699513024256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763165.982037:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763165.982043:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763165.982050:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513024256:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763167.005082:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763168.029182:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763169.053181:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763170.077183:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763171.101122:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763171.101148:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763171.101156:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763171.101160:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763171.101175:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3145 00000100:00080000:3.0:1789763171.101185:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763171.101192:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763171.101233:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699513024384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763171.101260:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763171.101263:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763171.101265:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763171.101268:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3145 00000100:00080000:3.0:1789763171.101272:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763171.101276:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763171.101301:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763171.101321:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699513024512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763171.101325:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513024384:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763171.101354:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763171.101357:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763171.101359:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763171.101365:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3145 00000100:00080000:3.0:1789763171.101368:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763171.101371:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763171.101379:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763171.101393:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699513024640/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763171.101403:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763171.101407:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763171.101410:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789763171.101416:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763171.101451:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763171/real 1789763171] req@ffff8dc0107ad880 x1876699513024384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763171.101466:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699513024384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763171.101474:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3145 00000100:00080000:3.0:1789763171.101486:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763171.101490:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763171.101493:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:0.0:1789763171.101499:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000008:00100000:3.0:1789763171.101502:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763171.101505:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513024384:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763171.101519:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699513024768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763171.101532:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513024512:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763171.101550:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763171.101560:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763171/real 1789763171] req@ffff8dc0107aea00 x1876699513024512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763171.101568:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699513024512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763171.101574:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763171.101576:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763171.101579:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513024512:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763171.101583:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513024640:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763171.101596:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763171.101603:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763171/real 1789763171] req@ffff8dc0107af100 x1876699513024640/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763171.101612:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699513024640/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763171.101617:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763171.101619:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763171.101621:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513024640:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763171.101692:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513024768:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763171.101724:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763171.101742:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763171/real 1789763171] req@ffff8dc0107ace00 x1876699513024768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763171.101751:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699513024768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763171.101760:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763171.101763:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763171.101770:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513024768:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763172.125109:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763173.149154:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763174.173147:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763175.197118:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763176.221080:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763176.221095:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763176.221101:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763176.221115:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3150 00000100:00080000:3.0:1789763176.221130:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763176.221140:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763176.221195:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699513024896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1789763176.221236:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763176.221237:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763176.221274:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763176.221279:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763176.221287:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3150 00000100:00080000:3.0:1789763176.221296:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763176.221304:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763176.221320:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763176.221330:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513024896:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763176.221347:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699513025024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763176.221364:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763176.221369:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763176.221372:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763176.221378:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3150 00000100:00080000:3.0:1789763176.221382:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763176.221388:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763176.221395:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763176.221411:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699513025152/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763176.221425:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763176.221430:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763176.221432:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763176.221436:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3150 00000800:00000100:0.0:1789763176.221446:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763176.221449:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763176.221459:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763176.221467:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763176.221483:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699513025280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763176.221525:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763176/real 1789763176] req@ffff8dc0107ae680 x1876699513024896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763176.221560:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699513024896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763176.221576:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763176.221582:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763176.221588:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513024896:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763176.221620:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513025024:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763176.221649:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763176.221665:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763176/real 1789763176] req@ffff8dc0107af800 x1876699513025024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763176.221681:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699513025024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763176.221691:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763176.221697:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763176.221701:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513025024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763176.221721:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513025152:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763176.221746:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763176.221760:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763176/real 1789763176] req@ffff8dc0107ac380 x1876699513025152/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763176.221774:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699513025152/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763176.221785:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763176.221789:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763176.221793:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513025152:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763176.221800:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513025280:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763176.221820:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763176.221831:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763176/real 1789763176] req@ffff8dc0107adf80 x1876699513025280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763176.221845:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699513025280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763176.221855:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763176.221859:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763176.221862:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513025280:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763177.245167:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763178.269190:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763179.293133:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763180.317252:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763181.341070:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763181.341087:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763181.341094:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763181.341110:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3155 00000100:00080000:3.0:1789763181.341140:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763181.341155:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789763181.341165:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763181.341260:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699513025408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763181.341312:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763181.341321:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763181.341328:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763181.341340:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3155 00000100:00080000:3.0:1789763181.341351:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763181.341361:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763181.341380:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513025408:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789763181.341384:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763181.341425:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699513025536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763181.341448:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763181.341453:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763181.341456:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763181.341465:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3155 00000100:00080000:3.0:1789763181.341473:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763181.341480:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763181.341488:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789763181.341500:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763181.341511:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699513025664/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763181.341533:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763181.341539:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763181.341541:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763181.341548:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3155 00000100:00080000:3.0:1789763181.341556:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763181.341564:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763181.341572:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763181.341583:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763181/real 1789763181] req@ffff8dc0107af480 x1876699513025408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763181.341593:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699513025792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763181.341655:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699513025408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763181.341695:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763181.341703:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763181.341711:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513025408:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763181.341747:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513025536:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763181.341776:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763181.341791:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763181/real 1789763181] req@ffff8dc0107ad180 x1876699513025536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763181.341808:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699513025536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763181.341818:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763181.341822:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763181.341827:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513025536:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763181.341834:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513025664:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763181.341855:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763181.341866:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763181/real 1789763181] req@ffff8dc0107ac700 x1876699513025664/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763181.341882:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699513025664/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763181.341893:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763181.341896:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763181.341933:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513025664:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763181.341942:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513025792:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763181.341967:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763181.341982:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763181/real 1789763181] req@ffff8dc0107aed80 x1876699513025792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763181.341997:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699513025792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763181.342007:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763181.342012:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763181.342016:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513025792:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763182.365104:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763183.389135:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763184.413098:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763185.437200:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763186.461096:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763186.461109:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763186.461115:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763186.461130:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3161 00010000:00010000:2.0:1789763186.461131:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763186.461145:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763186.461158:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763186.461237:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699513025920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763186.461306:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763186.461314:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763186.461318:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763186.461326:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3161 00000100:00080000:1.0:1789763186.461335:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763186.461345:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763186.461363:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763186.461375:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513025920:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763186.461393:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699513026048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763186.461409:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763186.461414:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763186.461417:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763186.461422:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3161 00000100:00080000:1.0:1789763186.461428:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763186.461433:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763186.461440:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763186.461456:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699513026176/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763186.461486:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789763186.461488:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789763186.461502:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763186.461506:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763186.461514:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3161 00000100:00080000:1.0:1789763186.461521:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763186.461527:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763186.461534:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763186.461549:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699513026304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763186.461577:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763186/real 1789763186] req@ffff8dc005c94380 x1876699513025920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763186.461620:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699513025920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763186.461644:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763186.461653:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763186.461662:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513025920:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763186.461703:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513026048:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763186.461753:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763186.461781:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763186/real 1789763186] req@ffff8dc005c95f80 x1876699513026048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763186.461804:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699513026048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763186.461815:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763186.461820:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763186.461824:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513026048:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763186.461833:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513026176:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763186.461863:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763186.461877:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763186/real 1789763186] req@ffff8dc005c94700 x1876699513026176/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763186.461892:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699513026176/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763186.461937:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763186.461942:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763186.461946:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513026176:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763186.461955:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513026304:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763186.461980:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763186.461995:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763186/real 1789763186] req@ffff8dc005c96a00 x1876699513026304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763186.462010:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699513026304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763186.462020:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763186.462024:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763186.462028:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513026304:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763187.485127:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763188.509186:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763189.533175:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763190.557104:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763191.581099:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763191.581113:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763191.581118:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763191.581155:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3166 00010000:00010000:1.0:1789763191.581164:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763191.581170:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763191.581182:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763191.581232:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699513026432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763191.581270:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763191.581275:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763191.581278:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763191.581284:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3166 00000100:00080000:3.0:1789763191.581288:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763191.581294:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763191.581308:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763191.581331:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513026432:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763191.581332:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699513026560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763191.581360:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763191.581364:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763191.581367:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763191.581374:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3166 00000100:00080000:3.0:1789763191.581379:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763191.581384:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763191.581392:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763191.581409:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699513026688/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763191.581423:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763191.581430:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763191.581434:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789763191.581444:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763191.581447:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3166 00000100:00080000:3.0:1789763191.581461:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763191.581472:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763191.581485:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763191.581505:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763191/real 1789763191] req@ffff8dc0107ad500 x1876699513026432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763191.581517:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699513026816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763191.581538:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699513026432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763191.581558:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763191.581564:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763191.581570:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513026432:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763191.581603:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513026560:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763191.581635:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763191.581651:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763191/real 1789763191] req@ffff8dc0107ac000 x1876699513026560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763191.581667:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699513026560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763191.581677:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763191.581681:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763191.581686:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513026560:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763191.581693:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513026688:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763191.581713:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763191.581724:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763191/real 1789763191] req@ffff8dc0107afb80 x1876699513026688/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763191.581738:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699513026688/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763191.581749:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763191.581752:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763191.581756:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513026688:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763191.581762:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513026816:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763191.581781:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763191.581792:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763191/real 1789763191] req@ffff8dc0107ae300 x1876699513026816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763191.581815:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699513026816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763191.581825:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763191.581828:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763191.581832:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513026816:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763192.605227:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763193.629115:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763194.653155:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763195.677111:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763196.701025:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763196.701039:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763196.701045:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763196.701060:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3171 00000100:00080000:3.0:1789763196.701075:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763196.701088:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789763196.701153:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763196.701161:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699513026944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763196.701210:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763196.701217:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763196.701220:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763196.701226:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3171 00000100:00080000:3.0:1789763196.701231:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763196.701237:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763196.701250:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763196.701272:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699513027072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763196.701289:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513026944:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763196.701296:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763196.701312:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763196.701315:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763196.701322:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3171 00000100:00080000:3.0:1789763196.701326:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763196.701330:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763196.701336:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763196.701358:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699513027200/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763196.701375:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763196.701381:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763196.701384:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763196.701391:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3171 00000100:00080000:3.0:1789763196.701396:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763196.701400:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763196.701407:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763196.701419:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699513027328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763196.701448:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763196.701529:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763196/real 1789763196] req@ffff8dc0107aca80 x1876699513026944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763196.701559:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699513026944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763196.701576:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763196.701583:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763196.701589:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513026944:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763196.701636:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513027072:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763196.701668:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763196.701685:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763196/real 1789763196] req@ffff8dc0107adc00 x1876699513027072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763196.701702:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699513027072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763196.701714:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763196.701719:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763196.701723:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513027072:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763196.701734:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513027200:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763196.701774:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763196.701798:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763196/real 1789763196] req@ffff8dc0107ae300 x1876699513027200/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763196.701824:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699513027200/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763196.701843:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763196.701850:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763196.701854:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513027200:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763196.701864:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513027328:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763196.701896:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763196.701952:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763196/real 1789763196] req@ffff8dc0107afb80 x1876699513027328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763196.701969:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699513027328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763196.701980:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763196.701984:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763196.701989:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513027328:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763197.725190:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763198.749161:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763199.773100:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763200.797175:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763201.821133:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763201.821149:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763201.821157:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763201.821174:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3176 00000100:00080000:3.0:1789763201.821192:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763201.821203:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763201.821226:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763201.821273:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699513027456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763201.821318:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763201.821323:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763201.821325:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763201.821331:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3176 00000100:00080000:3.0:1789763201.821336:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763201.821341:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763201.821356:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763201.821375:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699513027584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763201.821393:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763201.821400:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513027456:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763201.821400:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763201.821408:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763201.821427:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3176 00000100:00080000:3.0:1789763201.821433:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763201.821438:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763201.821444:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763201.821464:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699513027712/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763201.821476:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763201.821481:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763201.821484:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763201.821489:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3176 00000100:00080000:3.0:1789763201.821496:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763201.821501:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763201.821507:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763201.821528:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699513027840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763201.821542:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763201.821662:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763201/real 1789763201] req@ffff8dc0107ac000 x1876699513027456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763201.821691:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699513027456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763201.821706:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763201.821712:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763201.821717:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513027456:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763201.821752:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513027584:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763201.821800:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763201.821818:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763201/real 1789763201] req@ffff8dc0107ad500 x1876699513027584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763201.821834:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699513027584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763201.821846:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763201.821850:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763201.821854:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513027584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763201.821882:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513027712:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763201.821911:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763201.821977:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763201/real 1789763201] req@ffff8dc0107aed80 x1876699513027712/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763201.821994:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699513027712/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763201.822005:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763201.822009:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763201.822013:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513027712:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763201.822021:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513027840:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763201.822048:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763201.822110:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763201/real 1789763201] req@ffff8dc0107ac700 x1876699513027840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763201.822126:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699513027840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763201.822135:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763201.822139:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763201.822162:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513027840:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763202.845086:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763203.869207:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763204.893102:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763205.917144:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763206.941077:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763206.941093:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763206.941102:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763206.941120:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3181 00000100:00080000:3.0:1789763206.941139:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763206.941153:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789763206.941171:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763206.941211:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699513027968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763206.941252:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763206.941262:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763206.941265:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763206.941272:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3181 00000100:00080000:3.0:1789763206.941280:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763206.941289:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763206.941305:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763206.941317:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513027968:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763206.941334:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699513028096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763206.941366:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763206.941371:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763206.941375:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763206.941381:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3181 00000100:00080000:3.0:1789763206.941390:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763206.941396:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763206.941403:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789763206.941428:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763206.941429:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699513028224/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763206.941462:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763206.941467:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1789763206.941493:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763206/real 1789763206] req@ffff8dc0107ad180 x1876699513027968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763206.941524:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699513027968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763206.941542:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763206.941549:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763206.941557:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513027968:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763206.941591:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513028096:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1789763206.941608:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763206.941619:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3181 00000800:00000100:0.0:1789763206.941627:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763206.941641:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763206.941651:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763206.941660:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763206/real 1789763206] req@ffff8dc0107af480 x1876699513028096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789763206.941664:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763206.941684:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699513028096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763206.941695:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763206.941699:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763206.941703:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513028096:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763206.941711:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513028224:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763206.941744:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763206.941757:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763206/real 1789763206] req@ffff8dc0107adf80 x1876699513028224/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763206.941771:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699513028224/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763206.941791:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763206.941796:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763206.941802:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513028224:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789763206.941940:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699513028352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763206.942065:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513028352:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763206.942102:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763206.942129:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763206/real 1789763206] req@ffff8dc0107ac380 x1876699513028352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763206.942146:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699513028352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763206.942156:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763206.942160:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763206.942164:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513028352:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763207.965164:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763208.989110:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763210.013075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763211.037178:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763212.061119:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763212.061131:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:3.0:1789763212.061136:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763212.061138:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763212.061154:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3186 00000100:00080000:1.0:1789763212.061168:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763212.061179:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763212.061236:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699513028480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763212.061286:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763212.061293:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763212.061296:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763212.061302:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3186 00000100:00080000:1.0:1789763212.061310:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763212.061317:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763212.061330:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763212.061345:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513028480:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763212.061357:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699513028608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763212.061375:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763212.061380:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763212.061383:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763212.061388:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3186 00000100:00080000:1.0:1789763212.061392:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763212.061397:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763212.061403:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763212.061420:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699513028736/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763212.061435:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763212.061440:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763212.061443:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763212.061447:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3186 00000100:00080000:1.0:1789763212.061459:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763212.061464:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763212.061470:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763212.061483:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699513028864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763212.061551:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513028608:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763212.061608:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513028736:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763212.061638:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513028864:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763213.085101:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763214.109201:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763215.133140:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789763215.133400:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1789763215.133449:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789763215.133497:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789763215.133551:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789763215.133566:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789763215.133582:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:0.0:1789763215.133621:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763212/real 1789763215] req@ffff8dc005c95c00 x1876699513028608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763215.133673:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699513028608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763215.133693:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763215.133701:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763215.133708:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513028608:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763215.133720:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763212/real 1789763215] req@ffff8dc005c94a80 x1876699513028736/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763215.133740:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699513028736/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763215.133752:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763215.133758:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763215.133763:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513028736:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763215.133772:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763212/real 1789763215] req@ffff8dc005c97100 x1876699513028864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763215.133791:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699513028864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763215.133802:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763215.133807:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763215.133812:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513028864:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763215.133822:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763212/real 1789763215] req@ffff8dc005c97800 x1876699513028480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763215.133843:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699513028480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763215.133855:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763215.133860:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763215.133865:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513028480:192.168.203.156@tcp:250:kworker.0 00010000:00010000:2.0:1789763216.157203:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763217.181065:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763217.181082:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763217.181089:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763217.181106:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3191 00010000:00010000:2.0:1789763217.181131:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763217.181132:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763217.181154:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763217.181217:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699513028992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763217.181260:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763217.181267:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763217.181271:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763217.181279:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3191 00000100:00080000:1.0:1789763217.181288:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763217.181295:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763217.181312:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763217.181330:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513028992:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763217.181340:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699513029120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763217.181359:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763217.181364:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763217.181367:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763217.181374:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3191 00000100:00080000:1.0:1789763217.181381:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763217.181388:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763217.181396:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763217.181414:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699513029248/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763217.181424:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789763217.181433:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763217.181439:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763217.181442:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763217.181448:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3191 00000100:00080000:1.0:1789763217.181454:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763217.181460:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763217.181469:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763217.181484:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763217/real 1789763217] req@ffff8dc005c95180 x1876699513028992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763217.181486:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699513029376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763217.181519:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699513028992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763217.181538:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763217.181546:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763217.181554:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513028992:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763217.181594:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513029120:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763217.181642:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763217.181669:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763217/real 1789763217] req@ffff8dc005c97800 x1876699513029120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763217.181693:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699513029120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763217.181710:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763217.181718:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763217.181724:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513029120:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763217.181736:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513029248:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763217.181777:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763217.181800:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763217/real 1789763217] req@ffff8dc005c97100 x1876699513029248/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763217.181823:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699513029248/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763217.181838:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763217.181844:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763217.181851:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513029248:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763217.181862:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513029376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763217.181974:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763217.182002:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763217/real 1789763217] req@ffff8dc005c94a80 x1876699513029376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763217.182027:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699513029376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763217.182043:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763217.182049:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763217.182055:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513029376:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763218.205225:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763219.229206:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763220.253196:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763221.277208:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763222.301093:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763222.301110:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763222.301117:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763222.301135:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3196 00000100:00080000:1.0:1789763222.301153:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763222.301169:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789763222.301230:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763222.301244:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699513029504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763222.301284:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763222.301293:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763222.301295:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763222.301301:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3196 00000100:00080000:1.0:1789763222.301306:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763222.301312:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763222.301327:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763222.301345:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699513029632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763222.301363:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763222.301371:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513029504:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789763222.301372:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763222.301379:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763222.301403:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3196 00000100:00080000:1.0:1789763222.301409:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763222.301413:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763222.301421:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763222.301440:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699513029760/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763222.301454:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763222.301459:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763222.301462:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763222.301474:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3196 00000100:00080000:1.0:1789763222.301489:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763222.301496:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763222.301504:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763222.301516:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763222.301525:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699513029888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763222.301624:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763222/real 1789763222] req@ffff8dc005c95c00 x1876699513029504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763222.301657:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699513029504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763222.301675:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763222.301681:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763222.301688:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513029504:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763222.301728:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513029632:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763222.301776:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763222.301804:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763222/real 1789763222] req@ffff8dc005c96a00 x1876699513029632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763222.301832:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699513029632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763222.301849:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763222.301854:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763222.301859:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513029632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763222.301869:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513029760:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763222.301977:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763222.302007:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763222/real 1789763222] req@ffff8dc005c94700 x1876699513029760/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763222.302045:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699513029760/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763222.302057:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763222.302062:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763222.302066:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513029760:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763222.302075:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513029888:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763222.302109:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763222.302124:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763222/real 1789763222] req@ffff8dc005c95f80 x1876699513029888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763222.302139:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699513029888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763222.302150:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763222.302154:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763222.302158:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513029888:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763223.325165:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763224.349127:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763225.373159:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763226.397167:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763227.421089:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763227.421102:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763227.421107:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763227.421120:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3202 00000100:00080000:1.0:1789763227.421145:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763227.421155:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763227.421201:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699513030016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763227.421238:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763227.421243:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763227.421246:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763227.421251:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3202 00000100:00080000:1.0:1789763227.421255:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763227.421261:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763227.421273:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763227.421292:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699513030144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763227.421309:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513030016:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789763227.421315:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1789763227.421316:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763227.421328:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763227.421332:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763227.421338:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3202 00000100:00080000:1.0:1789763227.421346:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763227.421351:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763227.421358:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763227.421375:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699513030272/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763227.421389:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763227.421394:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763227.421397:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763227.421404:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3202 00000100:00080000:1.0:1789763227.421409:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763227.421415:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789763227.421424:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1789763227.421428:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763227.421450:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699513030400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763227.421489:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763227/real 1789763227] req@ffff8dc005c94380 x1876699513030016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763227.421517:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699513030016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763227.421541:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763227.421547:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763227.421552:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513030016:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763227.421583:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513030144:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763227.421608:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763227.421623:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763227/real 1789763227] req@ffff8dc005c97b80 x1876699513030144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763227.421638:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699513030144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763227.421648:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763227.421653:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763227.421657:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513030144:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763227.421664:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513030272:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763227.421685:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763227.421696:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763227/real 1789763227] req@ffff8dc005c96300 x1876699513030272/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763227.421710:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699513030272/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763227.421720:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763227.421724:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763227.421727:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513030272:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763227.421735:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513030400:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763227.421754:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763227.421775:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763227/real 1789763227] req@ffff8dc005c95500 x1876699513030400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763227.421789:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699513030400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763227.421798:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763227.421801:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763227.421805:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513030400:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763228.445202:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763229.469135:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763230.493044:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763231.517083:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763232.541062:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763232.541085:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763232.541091:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763232.541106:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3207 00000100:00080000:1.0:1789763232.541121:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763232.541131:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789763232.541173:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763232.541194:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699513030528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763232.541233:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763232.541243:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763232.541245:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763232.541250:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3207 00000100:00080000:1.0:1789763232.541255:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763232.541261:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763232.541274:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763232.541297:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513030528:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763232.541311:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699513030656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763232.541329:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763232.541334:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763232.541337:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763232.541342:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3207 00000100:00080000:1.0:1789763232.541347:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763232.541352:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763232.541359:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763232.541375:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699513030784/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763232.541389:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789763232.541397:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789763232.541408:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763232.541414:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763232.541422:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3207 00000100:00080000:1.0:1789763232.541429:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763232.541434:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763232.541443:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763232.541458:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699513030912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763232.541474:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763232/real 1789763232] req@ffff8dc005c94e00 x1876699513030528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763232.541505:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699513030528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763232.541530:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763232.541536:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763232.541541:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513030528:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763232.541571:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513030656:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763232.541598:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763232.541615:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763232/real 1789763232] req@ffff8dc005c97480 x1876699513030656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763232.541631:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699513030656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763232.541641:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763232.541645:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763232.541649:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513030656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763232.541656:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513030784:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763232.541677:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763232.541688:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763232/real 1789763232] req@ffff8dc005c96680 x1876699513030784/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763232.541702:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699513030784/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763232.541712:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763232.541716:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763232.541720:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513030784:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763232.541726:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513030912:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763232.541745:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763232.541766:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763232/real 1789763232] req@ffff8dc005c95880 x1876699513030912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763232.541780:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699513030912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763232.541799:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763232.541803:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763232.541807:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513030912:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763233.565093:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763234.589140:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763235.613202:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763236.637177:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763237.661033:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763237.661049:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763237.661057:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763237.661076:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3212 00010000:00010000:3.0:1789763237.661086:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763237.661095:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763237.661113:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763237.661192:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96d80 x1876699513031040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763237.661246:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763237.661254:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763237.661258:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763237.661269:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3212 00000100:00080000:1.0:1789763237.661279:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763237.661290:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763237.661312:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513031040:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1789763237.661312:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763237.661356:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699513031168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763237.661375:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763237.661381:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763237.661383:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763237.661390:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3212 00000100:00080000:1.0:1789763237.661414:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763237.661420:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763237.661431:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789763237.661432:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763237.661457:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699513031296/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763237.661473:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763237.661478:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763237.661481:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763237.661486:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3212 00000100:00080000:1.0:1789763237.661491:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763237.661497:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763237.661504:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763237.661520:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763237/real 1789763237] req@ffff8dc005c96d80 x1876699513031040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763237.661534:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699513031424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763237.661590:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96d80 x1876699513031040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763237.661608:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763237.661615:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763237.661620:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513031040:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763237.661657:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513031168:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763237.661692:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763237.661710:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763237/real 1789763237] req@ffff8dc005c95880 x1876699513031168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763237.661726:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699513031168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763237.661737:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763237.661742:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763237.661746:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513031168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763237.661755:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513031296:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763237.661777:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763237.661788:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763237/real 1789763237] req@ffff8dc005c96680 x1876699513031296/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763237.661833:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699513031296/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763237.661845:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763237.661849:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763237.661853:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513031296:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763237.661861:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513031424:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763237.661890:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763237.661938:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763237/real 1789763237] req@ffff8dc005c97480 x1876699513031424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763237.661954:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699513031424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763237.661965:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763237.661969:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763237.661973:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513031424:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763238.685157:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763239.709036:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763240.733133:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763241.757159:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763242.781087:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763242.781104:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763242.781112:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763242.781129:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3217 00000100:00080000:2.0:1789763242.781144:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763242.781156:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763242.781223:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699513031552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1789763242.781245:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763242.781267:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763242.781275:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763242.781278:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763242.781284:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3217 00000100:00080000:2.0:1789763242.781290:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763242.781297:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763242.781312:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763242.781326:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513031552:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763242.781340:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699513031680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763242.781366:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763242.781371:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763242.781374:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763242.781380:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3217 00000100:00080000:2.0:1789763242.781384:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763242.781390:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763242.781399:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763242.781418:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699513031808/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763242.781432:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763242.781438:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763242.781441:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763242.781448:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3217 00000100:00080000:2.0:1789763242.781453:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763242.781460:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:3.0:1789763242.781470:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:2.0:1789763242.781472:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763242.781492:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699513031936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763242.781563:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763242/real 1789763242] req@ffff8dc00368d880 x1876699513031552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763242.781606:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699513031552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763242.781624:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763242.781632:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763242.781642:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513031552:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763242.781856:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513031680:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763242.782135:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763242.782164:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763242/real 1789763242] req@ffff8dc00368fb80 x1876699513031680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763242.782189:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699513031680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763242.782202:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763242.782210:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763242.782218:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513031680:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789763242.782235:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513031808:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789763242.782412:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763242.782438:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763242/real 1789763242] req@ffff8dc00368f480 x1876699513031808/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763242.782456:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699513031808/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763242.782468:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763242.782472:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763242.782477:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513031808:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789763242.782485:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513031936:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763242.782523:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763242.782548:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763242/real 1789763242] req@ffff8dc00368f100 x1876699513031936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763267 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763242.782574:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699513031936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763267 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763242.782592:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763242.782598:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763242.782603:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513031936:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763243.805075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763244.829182:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763245.853176:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763246.877165:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763247.901097:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1789763247.901108:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763247.901120:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763247.901127:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763247.901147:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3222 00000100:00080000:1.0:1789763247.901163:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763247.901175:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763247.901260:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699513032064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763247.901301:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763247.901310:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763247.901313:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763247.901318:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3222 00000100:00080000:1.0:1789763247.901326:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763247.901334:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763247.901349:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763247.901375:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513032064:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763247.901376:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699513032192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763247.901405:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763247.901410:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763247.901413:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763247.901419:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3222 00000100:00080000:1.0:1789763247.901424:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763247.901431:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763247.901441:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763247.901463:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699513032320/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763247.901483:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1789763247.901489:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789763247.901498:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763247.901503:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763247.901514:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3222 00000100:00080000:1.0:1789763247.901524:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763247.901534:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763247.901546:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763247.901568:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763247/real 1789763247] req@ffff8dc005c94e00 x1876699513032064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763247.901579:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699513032448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763247.901603:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699513032064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763247.901621:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763247.901626:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763247.901632:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513032064:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763247.901678:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513032192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763247.901704:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763247.901741:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763247/real 1789763247] req@ffff8dc005c95500 x1876699513032192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763247.901757:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699513032192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763247.901776:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763247.901780:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763247.901784:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513032192:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789763247.901791:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513032320:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789763247.901818:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763247.901831:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763247/real 1789763247] req@ffff8dc005c96300 x1876699513032320/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763247.901844:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699513032320/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763247.901853:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763247.901857:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763247.901861:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513032320:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789763247.901866:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513032448:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763247.901884:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763247.901894:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763247/real 1789763247] req@ffff8dc005c97b80 x1876699513032448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763247.901939:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699513032448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763247.901949:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763247.901953:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763247.901957:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513032448:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763248.925142:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763249.949058:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763250.973091:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763251.997148:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763253.021177:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763253.021196:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763253.021203:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763253.021228:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3227 00000100:00080000:3.0:1789763253.021246:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763253.021261:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763253.021297:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763253.021324:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699513032576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763253.021367:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763253.021373:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763253.021376:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763253.021381:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3227 00000100:00080000:3.0:1789763253.021387:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763253.021392:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763253.021405:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763253.021423:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699513032704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763253.021438:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763253.021443:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763253.021447:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763253.021459:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3227 00000100:00100000:0.0:1789763253.021463:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513032576:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763253.021467:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763253.021487:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763253.021496:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763253.021517:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699513032832/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763253.021530:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763253.021537:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763253.021540:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763253.021547:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3227 00000100:00080000:3.0:1789763253.021551:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763253.021558:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763253.021566:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763253.021581:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699513032960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763253.021600:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763253.021704:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763253/real 1789763253] req@ffff8dc0107af800 x1876699513032576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763253.021739:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699513032576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763253.021757:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763253.021764:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763253.021770:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513032576:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763253.021809:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513032704:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763253.021843:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763253.021859:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763253/real 1789763253] req@ffff8dc0107ae680 x1876699513032704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763253.021880:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699513032704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763253.021940:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763253.021949:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763253.021957:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513032704:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763253.021971:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513032832:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763253.022015:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763253.022039:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763253/real 1789763253] req@ffff8dc0107ace00 x1876699513032832/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763253.022065:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699513032832/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763253.022094:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763253.022102:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763253.022109:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513032832:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763253.022122:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513032960:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763253.022167:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763253.022191:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763253/real 1789763253] req@ffff8dc0107af100 x1876699513032960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763253.022215:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699513032960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763253.022226:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763253.022231:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763253.022235:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513032960:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763254.045187:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763255.069120:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763256.093195:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763257.117154:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763258.141099:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763258.141120:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763258.141128:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763258.141149:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3232 00000100:00080000:1.0:1789763258.141169:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763258.141181:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789763258.141208:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763258.141245:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699513033088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763258.141286:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763258.141291:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763258.141294:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763258.141299:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3232 00000100:00080000:1.0:1789763258.141304:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763258.141308:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763258.141323:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763258.141340:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699513033216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763258.141359:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763258.141364:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763258.141368:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763258.141377:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3232 00000100:00100000:0.0:1789763258.141383:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513033088:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789763258.141383:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763258.141395:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763258.141407:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763258.141422:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699513033344/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763258.141436:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763258.141441:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763258.141444:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763258.141450:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3232 00000100:00080000:1.0:1789763258.141453:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763258.141458:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763258.141465:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763258.141478:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699513033472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763258.141509:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763258.141584:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763258/real 1789763258] req@ffff8dc005c94380 x1876699513033088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763258.141617:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699513033088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763258.141634:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763258.141639:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763258.141678:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513033088:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763258.141715:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513033216:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763258.141750:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763258.141768:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763258/real 1789763258] req@ffff8dc005c95f80 x1876699513033216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763258.141784:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699513033216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763258.141795:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763258.141800:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763258.141804:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513033216:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763258.141811:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513033344:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763258.141833:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763258.141845:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763258/real 1789763258] req@ffff8dc005c94700 x1876699513033344/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763258.141859:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699513033344/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763258.141870:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763258.141874:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763258.141878:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513033344:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763258.141884:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513033472:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763258.141941:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763258.141957:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763258/real 1789763258] req@ffff8dc005c96a00 x1876699513033472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763258.141972:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699513033472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763258.141983:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763258.141987:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763258.141991:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513033472:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763259.165077:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763260.189189:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763261.213169:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763262.237127:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763263.261148:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763263.261165:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763263.261171:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:2.0:1789763263.261189:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763263.261192:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3237 00000100:00080000:3.0:1789763263.261213:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763263.261226:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763263.261299:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699513033600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763263.261363:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763263.261375:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763263.261379:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763263.261388:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3237 00000100:00080000:3.0:1789763263.261396:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763263.261405:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763263.261424:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763263.261455:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699513033728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763263.261471:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513033600:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763263.261480:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763263.261487:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763263.261490:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763263.261497:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3237 00000100:00080000:3.0:1789763263.261502:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763263.261507:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763263.261516:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763263.261538:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699513033856/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763263.261552:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763263.261557:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763263.261560:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763263.261565:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3237 00000100:00080000:3.0:1789763263.261573:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763263.261577:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763263.261585:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763263.261593:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763263.261629:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699513033984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763263.261705:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763263/real 1789763263] req@ffff8dc0107aea00 x1876699513033600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763263.261736:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699513033600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763263.261757:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763263.261764:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763263.261771:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513033600:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763263.261805:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513033728:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763263.261837:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763263.261855:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763263/real 1789763263] req@ffff8dc0107ad880 x1876699513033728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763263.261871:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699513033728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763263.261891:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763263.261895:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763263.261937:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513033728:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763263.261946:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513033856:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763263.261984:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763263.262000:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763263/real 1789763263] req@ffff8dc0107af100 x1876699513033856/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763263.262015:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699513033856/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763263.262027:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763263.262031:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763263.262035:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513033856:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763263.262043:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513033984:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763263.262065:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763263.262076:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763263/real 1789763263] req@ffff8dc0107ace00 x1876699513033984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763263.262091:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699513033984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763263.262100:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763263.262104:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763263.262109:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513033984:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763264.285543:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763265.309154:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763266.333189:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763267.357158:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763268.381034:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763268.381048:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763268.381054:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763268.381069:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3242 00000100:00080000:3.0:1789763268.381085:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763268.381096:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789763268.381115:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763268.381168:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699513034112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763268.381206:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763268.381211:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763268.381214:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763268.381220:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3242 00000100:00080000:3.0:1789763268.381224:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763268.381229:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763268.381242:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763268.381260:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699513034240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763268.381276:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763268.381280:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763268.381283:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763268.381288:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3242 00000100:00080000:3.0:1789763268.381296:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763268.381302:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513034112:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763268.381304:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763268.381320:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763268.381346:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699513034368/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763268.381361:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763268.381367:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763268.381370:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763268.381377:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3242 00000100:00080000:3.0:1789763268.381381:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763268.381386:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763268.381391:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763268.381404:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699513034496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763268.381482:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763268.381565:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763268/real 1789763268] req@ffff8dc0107ae680 x1876699513034112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763268.381635:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699513034112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763268.381653:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763268.381660:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763268.381666:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513034112:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763268.381706:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513034240:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763268.381742:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763268.381760:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763268/real 1789763268] req@ffff8dc0107af800 x1876699513034240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763268.381776:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699513034240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763268.381788:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763268.381792:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763268.381797:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513034240:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763268.381805:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513034368:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763268.381829:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763268.381841:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763268/real 1789763268] req@ffff8dc0107ac380 x1876699513034368/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763268.381856:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699513034368/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763268.381866:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763268.381870:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763268.381873:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513034368:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763268.381881:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513034496:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763268.381942:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763268.381958:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763268/real 1789763268] req@ffff8dc0107adf80 x1876699513034496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763268.381974:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699513034496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763268.381984:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763268.381989:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763268.381993:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513034496:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763269.405156:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763270.429114:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763271.453242:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763272.477106:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763273.501135:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763273.501152:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763273.501159:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763273.501177:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3248 00000100:00080000:1.0:1789763273.501213:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763273.501227:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789763273.501284:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763273.501325:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699513034624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763273.501386:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763273.501395:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763273.501400:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763273.501411:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3248 00000100:00080000:1.0:1789763273.501422:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763273.501435:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763273.501457:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513034624:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1789763273.501469:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763273.501514:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699513034752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763273.501548:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763273.501554:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763273.501557:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763273.501562:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3248 00000100:00080000:1.0:1789763273.501584:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763273.501590:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763273.501598:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789763273.501610:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763273.501624:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699513034880/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763273.501641:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763273.501646:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763273.501648:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763273.501654:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3248 00000100:00080000:1.0:1789763273.501660:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763273.501665:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763273.501674:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763273.501685:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763273/real 1789763273] req@ffff8dc005c95c00 x1876699513034624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763273.501692:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699513035008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763273.501810:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699513034624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763273.501827:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763273.501833:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763273.501844:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513034624:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763273.501879:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513034752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763273.501945:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763273.501963:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763273/real 1789763273] req@ffff8dc005c94a80 x1876699513034752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763273.501989:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699513034752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763273.502005:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763273.502012:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763273.502019:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513034752:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763273.502032:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513034880:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763273.502076:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763273.502098:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763273/real 1789763273] req@ffff8dc005c97100 x1876699513034880/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763273.502124:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699513034880/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763273.502143:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763273.502147:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763273.502151:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513034880:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763273.502159:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513035008:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763273.502185:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763273.502197:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763273/real 1789763273] req@ffff8dc005c97800 x1876699513035008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763273.502221:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699513035008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763273.502230:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763273.502234:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763273.502237:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513035008:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763274.525135:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763275.549125:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763276.573210:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763277.597029:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763278.621013:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763278.621028:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763278.621033:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763278.621050:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3253 00010000:00010000:2.0:1789763278.621055:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763278.621064:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763278.621074:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763278.621149:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699513035136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763278.621179:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763278.621184:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763278.621186:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763278.621190:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3253 00000100:00080000:3.0:1789763278.621194:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763278.621199:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763278.621226:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763278.621240:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699513035264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763278.621249:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763278.621252:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763278.621254:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763278.621259:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513035136:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763278.621260:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3253 00000100:00080000:3.0:1789763278.621264:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763278.621287:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763278.621293:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763278.621304:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699513035392/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763278.621312:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763278.621315:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763278.621316:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763278.621319:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3253 00000100:00080000:3.0:1789763278.621322:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763278.621325:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763278.621329:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763278.621339:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699513035520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763278.621416:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513035264:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763278.621444:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513035392:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763278.621467:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513035520:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763279.645113:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763280.669031:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763281.693121:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789763281.693319:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1789763281.693382:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789763281.693403:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789763281.693496:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789763281.693518:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789763281.693530:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:3.0:1789763281.693566:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763278/real 1789763281] req@ffff8dc0107ad180 x1876699513035264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763281.693601:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699513035264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763281.693615:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763281.693622:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763281.693629:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513035264:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789763281.693641:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763278/real 1789763281] req@ffff8dc0107ac700 x1876699513035392/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763281.693666:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699513035392/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763281.693678:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763281.693683:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763281.693688:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513035392:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789763281.693696:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763278/real 1789763281] req@ffff8dc0107aed80 x1876699513035520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763281.693714:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699513035520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763281.693730:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763281.693735:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763281.693739:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513035520:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789763281.693748:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763278/real 1789763281] req@ffff8dc0107af480 x1876699513035136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763281.693765:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699513035136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763281.693775:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763281.693779:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763281.693782:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513035136:192.168.203.156@tcp:250:kworker.0 00010000:00010000:2.0:1789763282.717133:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763283.741003:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763283.741046:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763283.741053:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763283.741057:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763283.741068:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3258 00000100:00080000:1.0:1789763283.741078:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763283.741084:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763283.741121:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699513035648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763283.741145:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763283.741148:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763283.741149:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763283.741152:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3258 00000100:00080000:1.0:1789763283.741155:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763283.741158:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763283.741178:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763283.741190:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699513035776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763283.741195:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513035648:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789763283.741207:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763283.741210:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763283.741211:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763283.741214:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3258 00000100:00080000:1.0:1789763283.741216:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763283.741219:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763283.741224:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763283.741237:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699513035904/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763283.741248:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763283.741252:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763283.741254:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763283.741258:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3258 00000100:00080000:1.0:1789763283.741263:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763283.741267:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:3.0:1789763283.741272:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1789763283.741273:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763283.741289:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699513036032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763283.741334:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763283/real 1789763283] req@ffff8dc005c95180 x1876699513035648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763283.741357:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699513035648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763283.741369:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763283.741374:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763283.741389:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513035648:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763283.741416:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513035776:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763283.741442:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763283.741471:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763283/real 1789763283] req@ffff8dc005c97800 x1876699513035776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763283.741486:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699513035776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763283.741495:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763283.741499:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763283.741502:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513035776:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789763283.741509:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513035904:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789763283.741532:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763283.741544:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763283/real 1789763283] req@ffff8dc005c97100 x1876699513035904/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763283.741555:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699513035904/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763283.741563:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763283.741566:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763283.741569:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513035904:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789763283.741574:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513036032:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763283.741595:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763283.741606:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763283/real 1789763283] req@ffff8dc005c94a80 x1876699513036032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763308 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763283.741622:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699513036032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763308 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763283.741629:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763283.741632:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763283.741634:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513036032:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763284.765060:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763285.789117:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763286.813070:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763287.837039:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763288.861010:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763288.861018:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763288.861022:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763288.861033:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3263 00000100:00080000:2.0:1789763288.861044:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1789763288.861052:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763288.861054:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763288.861096:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699513036160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763288.861126:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763288.861130:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763288.861132:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763288.861136:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3263 00000100:00080000:2.0:1789763288.861140:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763288.861144:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763288.861153:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763288.861167:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699513036288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763288.861177:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763288.861180:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763288.861182:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763288.861186:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3263 00000100:00080000:2.0:1789763288.861189:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763288.861193:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763288.861199:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763288.861209:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513036160:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763288.861218:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699513036416/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763288.861234:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763288.861238:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763288.861241:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763288.861244:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3263 00000100:00080000:2.0:1789763288.861247:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763288.861251:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763288.861255:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763288.861266:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699513036544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763288.861293:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763288.861346:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763288/real 1789763288] req@ffff8dc00368ce00 x1876699513036160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763288.861365:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699513036160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763288.861375:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763288.861378:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763288.861381:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513036160:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763288.861387:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513036288:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763288.861403:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763288.861412:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763288/real 1789763288] req@ffff8dc00368c000 x1876699513036288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763288.861438:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699513036288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763288.861448:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763288.861451:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763288.861453:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513036288:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789763288.861477:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513036416:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789763288.861494:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763288.861502:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763288/real 1789763288] req@ffff8dc00368ea00 x1876699513036416/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763288.861510:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699513036416/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763288.861515:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763288.861517:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763288.861519:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513036416:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789763288.861523:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513036544:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763288.861533:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763288.861539:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763288/real 1789763288] req@ffff8dc00368ca80 x1876699513036544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763288.861546:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699513036544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763288.861551:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763288.861553:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763288.861555:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513036544:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763289.885037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763290.909128:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763291.933146:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763292.957080:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763293.980985:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763293.980999:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763293.981004:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763293.981019:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3268 00000100:00080000:3.0:1789763293.981031:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763293.981038:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763293.981046:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763293.981082:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699513036672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763293.981107:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763293.981110:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763293.981111:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763293.981120:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3268 00000100:00080000:3.0:1789763293.981123:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763293.981126:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763293.981135:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763293.981143:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513036672:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763293.981148:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699513036800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763293.981156:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763293.981159:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763293.981160:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763293.981163:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3268 00000100:00080000:3.0:1789763293.981166:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763293.981169:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763293.981173:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763293.981181:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699513036928/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763293.981188:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763293.981191:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763293.981192:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763293.981194:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3268 00000100:00080000:3.0:1789763293.981196:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763293.981199:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763293.981201:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763293.981208:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699513037056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763293.981221:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763293.981266:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763293/real 1789763293] req@ffff8dc0107af480 x1876699513036672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763293.981282:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699513036672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763293.981291:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763293.981295:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763293.981298:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513036672:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763293.981319:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513036800:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763293.981334:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763293.981342:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763293/real 1789763293] req@ffff8dc0107aed80 x1876699513036800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763293.981350:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699513036800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763293.981361:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763293.981363:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763293.981366:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513036800:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763293.981369:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513036928:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763293.981395:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763293.981404:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763293/real 1789763293] req@ffff8dc0107ac700 x1876699513036928/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763293.981412:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699513036928/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763293.981418:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763293.981420:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763293.981422:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513036928:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763293.981425:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513037056:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763293.981437:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763293.981443:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763293/real 1789763293] req@ffff8dc0107ad180 x1876699513037056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763293.981451:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699513037056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763293.981456:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763293.981457:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763293.981460:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513037056:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763295.005102:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763296.029039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763297.053045:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763298.077108:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763299.101061:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763299.101081:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763299.101085:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763299.101097:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3273 00000100:00080000:3.0:1789763299.101108:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763299.101115:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789763299.101149:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763299.101165:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699513037184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763299.101191:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763299.101195:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763299.101208:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763299.101211:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3273 00000100:00080000:3.0:1789763299.101214:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763299.101216:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763299.101224:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763299.101234:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699513037312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763299.101243:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763299.101247:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763299.101249:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763299.101253:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513037184:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763299.101255:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3273 00000100:00080000:3.0:1789763299.101260:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763299.101291:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763299.101298:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763299.101314:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699513037440/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763299.101325:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763299.101328:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763299.101331:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763299.101337:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3273 00000100:00080000:3.0:1789763299.101357:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763299.101361:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763299.101376:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763299.101388:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699513037568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763299.101421:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763299.101477:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763299/real 1789763299] req@ffff8dc0107ad500 x1876699513037184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763299.101497:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699513037184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763299.101507:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763299.101511:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763299.101514:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513037184:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763299.101538:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513037312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763299.101561:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763299.101575:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763299/real 1789763299] req@ffff8dc0107ac000 x1876699513037312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763299.101590:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699513037312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763299.101599:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763299.101603:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763299.101605:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513037312:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763299.101611:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513037440:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763299.101630:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763299.101638:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763299/real 1789763299] req@ffff8dc0107afb80 x1876699513037440/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763299.101647:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699513037440/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763299.101652:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763299.101655:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763299.101657:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513037440:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763299.101661:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513037568:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763299.101672:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763299.101678:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763299/real 1789763299] req@ffff8dc0107ae300 x1876699513037568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763299.101686:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699513037568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763299.101691:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763299.101693:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763299.101695:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513037568:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763300.125088:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763301.149115:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763302.173055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763303.197050:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763304.220994:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763304.221004:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763304.221009:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763304.221020:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3278 00010000:00010000:2.0:1789763304.221025:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763304.221030:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763304.221038:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763304.221079:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699513037696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763304.221103:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763304.221106:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763304.221108:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763304.221111:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3278 00000100:00080000:1.0:1789763304.221114:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763304.221117:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763304.221126:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763304.221138:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513037696:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763304.221139:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699513037824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763304.221153:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763304.221155:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763304.221157:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763304.221160:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3278 00000100:00080000:1.0:1789763304.221163:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763304.221165:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763304.221169:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763304.221176:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699513037952/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763304.221184:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763304.221187:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763304.221189:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763304.221192:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3278 00000100:00080000:1.0:1789763304.221201:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763304.221204:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763304.221207:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763304.221214:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699513038080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763304.221224:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763304.221269:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763304/real 1789763304] req@ffff8dc005c95c00 x1876699513037696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763304.221290:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699513037696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763304.221300:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763304.221314:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763304.221319:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513037696:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763304.221346:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513037824:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763304.221369:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763304.221380:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763304/real 1789763304] req@ffff8dc005c96a00 x1876699513037824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763304.221389:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699513037824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763304.221396:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763304.221398:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763304.221400:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513037824:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763304.221405:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513037952:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763304.221418:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763304.221425:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763304/real 1789763304] req@ffff8dc005c94700 x1876699513037952/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763304.221433:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699513037952/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763304.221442:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763304.221445:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763304.221447:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513037952:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763304.221450:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513038080:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763304.221461:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763304.221466:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763304/real 1789763304] req@ffff8dc005c95f80 x1876699513038080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763304.221474:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699513038080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763304.221479:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763304.221481:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763304.221483:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513038080:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763305.244995:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763306.269031:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763307.293059:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763308.317009:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763309.341000:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763309.341010:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763309.341014:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763309.341032:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3283 00010000:00010000:1.0:1789763309.341035:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763309.341043:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763309.341052:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763309.341110:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699513038208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763309.341135:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763309.341137:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763309.341139:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763309.341142:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3283 00000100:00080000:3.0:1789763309.341145:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763309.341148:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763309.341158:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763309.341170:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513038208:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763309.341171:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699513038336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763309.341185:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763309.341188:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763309.341189:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763309.341191:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3283 00000100:00080000:3.0:1789763309.341193:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763309.341197:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763309.341201:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763309.341212:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699513038464/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763309.341219:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763309.341222:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763309.341223:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763309.341226:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3283 00000100:00080000:3.0:1789763309.341229:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763309.341231:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763309.341234:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763309.341242:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699513038592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763309.341258:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763309.341330:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763309/real 1789763309] req@ffff8dc0107adc00 x1876699513038208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763309.341346:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699513038208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763309.341355:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763309.341359:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763309.341363:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513038208:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763309.341384:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513038336:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763309.341401:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763309.341410:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763309/real 1789763309] req@ffff8dc0107aca80 x1876699513038336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763309.341418:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699513038336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763309.341424:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763309.341426:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763309.341429:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513038336:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763309.341432:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513038464:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763309.341443:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763309.341449:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763309/real 1789763309] req@ffff8dc0107ae300 x1876699513038464/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763309.341456:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699513038464/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763309.341461:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763309.341463:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763309.341465:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513038464:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763309.341468:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513038592:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763309.341481:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763309.341487:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763309/real 1789763309] req@ffff8dc0107afb80 x1876699513038592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763309.341494:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699513038592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763309.341499:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763309.341501:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763309.341503:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513038592:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763310.365078:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763311.389097:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763312.413087:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763313.437055:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763314.461102:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763314.461116:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763314.461122:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763314.461136:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3289 00000100:00080000:3.0:1789763314.461150:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763314.461160:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763314.461210:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699513038720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763314.461276:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763314.461283:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763314.461287:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763314.461295:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3289 00000100:00080000:3.0:1789763314.461301:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763314.461307:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763314.461322:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00010000:00010000:0.0:1789763314.461331:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763314.461345:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699513038848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763314.461366:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763314.461370:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513038720:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763314.461375:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763314.461383:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763314.461392:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3289 00000100:00080000:3.0:1789763314.461397:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763314.461403:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763314.461412:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763314.461446:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699513038976/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763314.461462:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763314.461468:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763314.461474:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789763314.461475:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763314.461491:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3289 00000100:00080000:3.0:1789763314.461500:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763314.461506:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763314.461514:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763314.461544:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763314/real 1789763314] req@ffff8dc0107ac000 x1876699513038720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763314.461548:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699513039104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763314.461615:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699513038720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763314.461640:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763314.461648:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763314.461656:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513038720:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763314.461694:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513038848:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763314.461737:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763314.461760:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763314/real 1789763314] req@ffff8dc0107ad500 x1876699513038848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763314.461778:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699513038848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763314.461790:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763314.461794:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763314.461798:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513038848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763314.461806:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513038976:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763314.461829:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763314.461841:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763314/real 1789763314] req@ffff8dc0107ad180 x1876699513038976/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763314.461855:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699513038976/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763314.461867:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763314.461871:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763314.461875:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513038976:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763314.461882:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513039104:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763314.461953:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763314.461976:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763314/real 1789763314] req@ffff8dc0107ac700 x1876699513039104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763314.461997:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699513039104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763314.462017:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763314.462022:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763314.462029:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513039104:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763315.485163:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763316.509104:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763317.533175:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763318.557183:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763319.581115:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763319.581134:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:1.0:1789763319.581137:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763319.581157:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763319.581181:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3294 00000100:00080000:3.0:1789763319.581221:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763319.581235:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763319.581295:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699513039232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763319.581331:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763319.581335:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763319.581338:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763319.581342:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3294 00000100:00080000:3.0:1789763319.581347:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763319.581352:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763319.581365:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763319.581385:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699513039360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763319.581386:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513039232:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763319.581408:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763319.581412:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763319.581413:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763319.581417:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3294 00000100:00080000:3.0:1789763319.581420:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763319.581425:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763319.581431:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763319.581445:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699513039488/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763319.581456:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763319.581460:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763319.581462:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763319.581466:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3294 00000100:00080000:3.0:1789763319.581469:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763319.581473:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763319.581478:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763319.581490:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699513039616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763319.581501:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763319.581561:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763319/real 1789763319] req@ffff8dc0107aed80 x1876699513039232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763319.581584:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699513039232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763319.581597:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763319.581601:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763319.581606:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513039232:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763319.581635:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513039360:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763319.581658:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763319.581671:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763319/real 1789763319] req@ffff8dc0107af480 x1876699513039360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763319.581684:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699513039360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763319.581693:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763319.581696:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763319.581701:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513039360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763319.581707:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513039488:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763319.581725:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763319.581734:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763319/real 1789763319] req@ffff8dc0107adf80 x1876699513039488/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763319.581746:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699513039488/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763319.581754:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763319.581757:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763319.581760:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513039488:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763319.581766:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513039616:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763319.581801:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763319.581813:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763319/real 1789763319] req@ffff8dc0107ac380 x1876699513039616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763319.581826:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699513039616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763319.581835:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763319.581838:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763319.581842:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513039616:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763320.605107:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763321.629135:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763322.653186:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763323.677082:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763324.701086:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:0.0:1789763324.701101:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763324.701103:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763324.701111:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763324.701129:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3299 00000100:00080000:2.0:1789763324.701145:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763324.701188:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763324.701252:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699513039744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763324.701295:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763324.701300:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763324.701303:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763324.701309:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3299 00000100:00080000:2.0:1789763324.701315:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763324.701320:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763324.701334:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763324.701353:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699513039872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763324.701369:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763324.701375:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789763324.701386:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513039744:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763324.701387:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763324.701411:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3299 00000100:00080000:2.0:1789763324.701416:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763324.701421:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763324.701430:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763324.701451:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699513040000/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763324.701464:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763324.701469:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763324.701471:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763324.701475:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3299 00000100:00080000:2.0:1789763324.701481:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763324.701485:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763324.701491:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763324.701504:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699513040128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763324.701537:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763324.701631:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763324/real 1789763324] req@ffff8dc00368ed80 x1876699513039744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763324.701670:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699513039744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763324.701694:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763324.701702:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763324.701710:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513039744:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763324.701755:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513039872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763324.701804:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763324.701832:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763324/real 1789763324] req@ffff8dc00368e300 x1876699513039872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763324.701859:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699513039872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763324.701876:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763324.701883:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763324.701891:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513039872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763324.701952:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513040000:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763324.701998:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763324.702024:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763324/real 1789763324] req@ffff8dc00368f800 x1876699513040000/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763324.702048:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699513040000/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763324.702064:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763324.702071:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763324.702077:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513040000:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763324.702090:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513040128:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763324.702128:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763324.702149:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763324/real 1789763324] req@ffff8dc00368df80 x1876699513040128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763324.702207:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699513040128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763324.702224:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763324.702230:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763324.702237:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513040128:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763325.725165:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763326.749075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763327.773077:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763328.797090:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763329.821055:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763329.821127:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763329.821136:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763329.821141:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763329.821152:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3304 00000100:00080000:3.0:1789763329.821164:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763329.821170:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763329.821226:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699513040256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763329.821266:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763329.821271:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763329.821275:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763329.821281:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3304 00000100:00080000:3.0:1789763329.821287:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763329.821294:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763329.821307:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763329.821327:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513040256:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763329.821331:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699513040384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763329.821354:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763329.821357:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763329.821360:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763329.821365:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3304 00000100:00080000:3.0:1789763329.821369:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763329.821379:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763329.821385:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763329.821406:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699513040512/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763329.821420:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763329.821426:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763329.821428:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763329.821433:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3304 00000800:00000100:0.0:1789763329.821441:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763329.821442:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763329.821454:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763329.821461:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763329.821478:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699513040640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763329.821528:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763329/real 1789763329] req@ffff8dc0107af800 x1876699513040256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763329.821556:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699513040256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763329.821571:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763329.821578:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763329.821584:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513040256:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763329.821614:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513040384:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763329.821667:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763329.821676:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763329/real 1789763329] req@ffff8dc0107ae680 x1876699513040384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763329.821684:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699513040384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763329.821690:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763329.821692:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763329.821695:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513040384:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763329.821699:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513040512:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763329.821720:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763329.821726:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763329/real 1789763329] req@ffff8dc0107ace00 x1876699513040512/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763329.821734:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699513040512/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763329.821743:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763329.821745:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763329.821747:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513040512:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763329.821750:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513040640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763329.821760:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763329.821766:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763329/real 1789763329] req@ffff8dc0107af100 x1876699513040640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763329.821772:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699513040640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763329.821778:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763329.821780:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763329.821782:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513040640:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763330.845060:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763331.869081:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763332.893064:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763333.917063:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763334.941014:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763334.941027:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763334.941032:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789763334.941040:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763334.941041:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3309 00000100:00080000:3.0:1789763334.941057:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763334.941063:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763334.941112:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699513040768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763334.941136:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763334.941139:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763334.941140:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763334.941143:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3309 00000100:00080000:3.0:1789763334.941145:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763334.941149:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763334.941157:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763334.941167:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699513040896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763334.941174:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513040768:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763334.941177:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763334.941182:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763334.941183:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763334.941186:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3309 00000100:00080000:3.0:1789763334.941189:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763334.941192:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763334.941196:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763334.941206:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699513041024/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763334.941213:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763334.941215:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763334.941216:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763334.941219:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3309 00000100:00080000:3.0:1789763334.941222:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763334.941225:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763334.941229:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763334.941236:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763334.941238:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699513041152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763334.941281:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763334/real 1789763334] req@ffff8dc0107ad880 x1876699513040768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763334.941295:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699513040768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763334.941304:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763334.941307:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763334.941310:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513040768:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763334.941350:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513040896:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763334.941379:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763334.941405:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763334/real 1789763334] req@ffff8dc0107aea00 x1876699513040896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763334.941417:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699513040896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763334.941425:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763334.941428:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763334.941431:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513040896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763334.941437:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513041024:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763334.941452:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763334.941462:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763334/real 1789763334] req@ffff8dc0107af100 x1876699513041024/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763334.941476:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699513041024/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763334.941489:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763334.941494:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763334.941499:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513041024:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763334.941509:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513041152:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763334.941551:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763334.941564:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763334/real 1789763334] req@ffff8dc0107ace00 x1876699513041152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763334.941573:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699513041152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763334.941579:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763334.941582:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763334.941584:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513041152:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763335.965163:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763336.989151:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763338.013033:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763339.037008:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763340.061100:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763340.061113:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763340.061120:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763340.061135:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3314 00000100:00080000:3.0:1789763340.061149:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763340.061160:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789763340.061164:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763340.061215:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699513041280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763340.061255:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763340.061260:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763340.061264:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763340.061269:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3314 00000100:00080000:3.0:1789763340.061274:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763340.061279:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763340.061293:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763340.061310:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699513041408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763340.061329:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513041280:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763340.061330:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763340.061346:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763340.061356:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763340.061364:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3314 00000100:00080000:3.0:1789763340.061368:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763340.061371:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763340.061379:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763340.061399:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699513041536/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763340.061415:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763340.061421:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763340.061424:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763340.061431:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3314 00000100:00080000:3.0:1789763340.061436:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763340.061440:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763340.061447:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763340.061461:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699513041664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763340.061482:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763340.061581:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763340/real 1789763340] req@ffff8dc0107ae680 x1876699513041280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763340.061626:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699513041280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763340.061652:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763340.061661:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763340.061667:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513041280:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763340.061708:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513041408:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763340.061749:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763340.061767:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763340/real 1789763340] req@ffff8dc0107af800 x1876699513041408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763340.061784:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699513041408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763340.061795:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763340.061799:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763340.061804:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513041408:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763340.061813:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513041536:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763340.061853:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763340.061867:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763340/real 1789763340] req@ffff8dc0107ac380 x1876699513041536/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763340.061882:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699513041536/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763340.061892:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763340.061896:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763340.061935:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513041536:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763340.061943:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513041664:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763340.061974:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763340.061997:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763340/real 1789763340] req@ffff8dc0107adf80 x1876699513041664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763340.062035:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699513041664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763340.062080:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763340.062087:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763340.062094:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513041664:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763341.085170:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763342.109187:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763343.133075:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763344.157190:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763345.181071:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763345.181107:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763345.181114:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763345.181134:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3319 00000100:00080000:3.0:1789763345.181150:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763345.181163:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789763345.181207:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763345.181224:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699513041792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763345.181262:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763345.181267:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763345.181269:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763345.181274:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3319 00000100:00080000:3.0:1789763345.181281:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763345.181287:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763345.181300:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763345.181320:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513041792:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763345.181322:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699513041920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763345.181347:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763345.181350:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763345.181352:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763345.181358:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3319 00000100:00080000:3.0:1789763345.181362:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763345.181367:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763345.181374:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763345.181391:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699513042048/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763345.181403:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763345.181408:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763345.181411:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763345.181415:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3319 00000100:00080000:3.0:1789763345.181419:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763345.181424:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763345.181430:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763345.181446:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699513042176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763345.181492:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513041920:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763345.181552:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513042048:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763345.181581:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513042176:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763346.205103:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763347.229198:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763348.253267:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789763348.253279:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1789763348.253344:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789763348.253362:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789763348.253447:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789763348.253464:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789763348.253479:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1789763348.253521:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763345/real 1789763348] req@ffff8dc0107aed80 x1876699513041920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763348.253590:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699513041920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763348.253615:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763348.253622:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763348.253627:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513041920:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763348.253636:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763345/real 1789763348] req@ffff8dc0107ac700 x1876699513042048/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763348.253663:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699513042048/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763348.253674:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763348.253679:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763348.253683:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513042048:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763348.253692:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763345/real 1789763348] req@ffff8dc0107ad180 x1876699513042176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763348.253708:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699513042176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763348.253719:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763348.253724:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763348.253728:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513042176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763348.253736:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763345/real 1789763348] req@ffff8dc0107af480 x1876699513041792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763348.253757:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699513041792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763348.253766:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763348.253770:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763348.253774:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513041792:192.168.203.156@tcp:250:kworker.0 00010000:00010000:0.0:1789763349.277125:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763350.301079:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763350.301092:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763350.301098:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763350.301115:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3324 00010000:00010000:2.0:1789763350.301128:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763350.301133:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763350.301147:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763350.301203:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699513042304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763350.301246:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763350.301252:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763350.301256:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763350.301266:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3324 00000100:00080000:3.0:1789763350.301274:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763350.301283:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763350.301310:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763350.301343:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699513042432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763350.301362:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513042304:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763350.301372:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763350.301388:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763350.301394:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763350.301404:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3324 00000100:00080000:3.0:1789763350.301413:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763350.301419:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763350.301436:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763350.301457:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699513042560/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763350.301477:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763350.301482:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763350.301486:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763350.301497:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3324 00000100:00080000:3.0:1789763350.301503:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763350.301508:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763350.301515:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789763350.301529:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763350.301532:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699513042688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763350.301651:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763350/real 1789763350] req@ffff8dc0107ad500 x1876699513042304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763350.301686:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699513042304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763350.301707:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763350.301715:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763350.301723:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513042304:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763350.301768:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513042432:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763350.301815:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763350.301838:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763350/real 1789763350] req@ffff8dc0107ac000 x1876699513042432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763350.301862:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699513042432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763350.301877:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763350.301884:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763350.301890:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513042432:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763350.301941:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513042560:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789763350.302023:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763350.302048:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763350/real 1789763350] req@ffff8dc0107afb80 x1876699513042560/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763350.302073:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699513042560/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763350.302089:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763350.302096:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763350.302102:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513042560:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763350.302115:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513042688:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763350.302153:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763350.302175:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763350/real 1789763350] req@ffff8dc0107ae300 x1876699513042688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763350.302198:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699513042688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763350.302213:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763350.302219:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763350.302225:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513042688:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763351.325269:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763352.349057:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763353.373189:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763354.397162:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763355.421060:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763355.421074:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763355.421085:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763355.421088:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763355.421097:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3330 00000100:00080000:2.0:1789763355.421106:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763355.421112:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763355.421143:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699513042816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763355.421167:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763355.421169:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763355.421171:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763355.421174:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3330 00000100:00080000:2.0:1789763355.421176:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763355.421179:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763355.421187:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763355.421197:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699513042944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763355.421207:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763355.421213:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513042816:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763355.421213:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763355.421220:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763355.421224:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3330 00000100:00080000:2.0:1789763355.421232:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763355.421243:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763355.421247:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763355.421259:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699513043072/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763355.421266:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763355.421269:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763355.421271:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763355.421274:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3330 00000100:00080000:2.0:1789763355.421276:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763355.421279:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763355.421282:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763355.421290:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699513043200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763355.421317:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763355.421360:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763355/real 1789763355] req@ffff8dc00368dc00 x1876699513042816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763355.421375:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699513042816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763355.421384:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763355.421388:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763355.421391:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513042816:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763355.421414:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513042944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763355.421428:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763355.421437:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763355/real 1789763355] req@ffff8dc00368e680 x1876699513042944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763355.421444:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699513042944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763355.421450:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763355.421452:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763355.421454:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513042944:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763355.421459:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513043072:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763355.421470:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763355.421476:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763355/real 1789763355] req@ffff8dc00368d180 x1876699513043072/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763355.421488:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699513043072/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763355.421493:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763355.421495:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763355.421496:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513043072:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763355.421500:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513043200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763355.421511:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763355.421517:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763355/real 1789763355] req@ffff8dc00368c700 x1876699513043200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763355.421524:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699513043200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763355.421529:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763355.421531:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763355.421533:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513043200:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763356.445153:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763357.469182:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763358.493121:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763359.517144:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763360.541098:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763360.541112:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763360.541119:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763360.541136:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3335 00000100:00080000:3.0:1789763360.541152:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763360.541162:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763360.541214:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763360.541215:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699513043328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763360.541260:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763360.541266:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763360.541268:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763360.541274:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3335 00000100:00080000:3.0:1789763360.541278:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763360.541284:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763360.541297:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763360.541314:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699513043456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763360.541323:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513043328:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763360.541342:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763360.541349:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763360.541351:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763360.541358:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3335 00000100:00080000:3.0:1789763360.541362:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763360.541367:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763360.541374:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763360.541394:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699513043584/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763360.541409:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763360.541414:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763360.541416:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763360.541421:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3335 00000100:00080000:3.0:1789763360.541426:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789763360.541435:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763360.541440:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763360.541452:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763360.541477:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699513043712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763360.541498:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763360/real 1789763360] req@ffff8dc0107aca80 x1876699513043328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763360.541526:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699513043328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763360.541545:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763360.541551:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763360.541557:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513043328:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763360.541587:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513043456:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763360.541614:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763360.541629:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763360/real 1789763360] req@ffff8dc0107adc00 x1876699513043456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763360.541645:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699513043456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763360.541656:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763360.541660:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763360.541664:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513043456:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763360.541671:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513043584:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763360.541691:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763360.541702:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763360/real 1789763360] req@ffff8dc0107ae300 x1876699513043584/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763360.541716:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699513043584/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763360.541727:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763360.541730:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763360.541734:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513043584:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763360.541741:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513043712:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763360.541761:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763360.541772:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763360/real 1789763360] req@ffff8dc0107afb80 x1876699513043712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763360.541787:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699513043712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763360.541796:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763360.541800:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763360.541803:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513043712:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763361.565144:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763362.589144:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763363.613196:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763364.637087:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763365.661070:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763365.661086:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763365.661092:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763365.661106:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3340 00000100:00080000:3.0:1789763365.661120:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763365.661130:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763365.661195:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699513043840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763365.661224:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763365.661240:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763365.661243:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763365.661248:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3340 00000100:00080000:3.0:1789763365.661253:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763365.661258:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763365.661272:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763365.661287:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699513043968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763365.661300:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763365.661304:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763365.661306:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763365.661310:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3340 00000100:00080000:3.0:1789763365.661314:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763365.661319:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763365.661325:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763365.661341:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699513044096/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763365.661353:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763365.661357:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763365.661360:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763365.661364:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3340 00000100:00080000:3.0:1789763365.661368:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763365.661373:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763365.661378:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763365.661390:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699513044224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1789763365.661707:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763365.661748:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513043840:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1789763365.661844:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763365.661960:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763365/real 1789763365] req@ffff8dc0107ac000 x1876699513043840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763365.661983:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699513043840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763365.661994:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763365.661999:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763365.662003:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513043840:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763365.662012:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513043968:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763365.662037:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763365.662050:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763365/real 1789763365] req@ffff8dc0107ad500 x1876699513043968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763365.662062:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699513043968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763365.662071:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763365.662093:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763365.662100:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513043968:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763365.662113:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513044096:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763365.662151:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763365.662175:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763365/real 1789763365] req@ffff8dc0107af480 x1876699513044096/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763365.662202:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699513044096/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763365.662233:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763365.662248:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763365.662253:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513044096:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763365.662263:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513044224:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763365.662304:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763365.662316:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763365/real 1789763365] req@ffff8dc0107ad180 x1876699513044224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763365.662324:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699513044224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763365.662330:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763365.662332:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763365.662335:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513044224:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763366.685125:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763367.709074:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763368.733113:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763369.757145:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763370.781059:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763370.781076:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763370.781083:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789763370.781097:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763370.781107:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3345 00000100:00080000:3.0:1789763370.781128:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763370.781144:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763370.781220:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699513044352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763370.781264:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763370.781270:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763370.781272:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763370.781278:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3345 00000100:00080000:3.0:1789763370.781284:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763370.781290:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763370.781303:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763370.781322:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699513044480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763370.781335:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763370.781339:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763370.781342:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763370.781347:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3345 00000100:00080000:3.0:1789763370.781352:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763370.781358:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763370.781364:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763370.781376:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699513044608/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763370.781391:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763370.781396:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763370.781398:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763370.781406:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3345 00000100:00080000:3.0:1789763370.781413:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763370.781418:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763370.781424:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763370.781437:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513044352:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763370.781447:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699513044736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763370.781564:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763370.781649:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763370/real 1789763370] req@ffff8dc0107ac700 x1876699513044352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763370.781682:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699513044352/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763370.781697:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763370.781703:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763370.781708:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513044352:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763370.781718:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513044480:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763370.781748:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763370.781763:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763370/real 1789763370] req@ffff8dc0107aed80 x1876699513044480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763370.781777:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699513044480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763370.781787:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763370.781791:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763370.781795:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513044480:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763370.781805:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513044608:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763370.781861:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763370.781876:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763370/real 1789763370] req@ffff8dc0107adf80 x1876699513044608/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763370.781890:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699513044608/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763370.781961:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763370.781966:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763370.781970:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513044608:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763370.782005:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513044736:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763370.782030:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763370.782044:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763370/real 1789763370] req@ffff8dc0107ac380 x1876699513044736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763370.782058:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699513044736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763370.782068:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763370.782071:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763370.782075:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513044736:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763371.805084:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763372.829130:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763373.853079:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763374.877074:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763375.901050:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763375.901063:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763375.901071:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763375.901085:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3350 00000100:00080000:3.0:1789763375.901099:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763375.901110:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763375.901163:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699513044864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1789763375.901186:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763375.901206:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763375.901214:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763375.901219:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763375.901226:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3350 00000100:00080000:3.0:1789763375.901232:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763375.901239:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763375.901257:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763375.901265:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513044864:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763375.901289:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699513044992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763375.901315:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763375.901323:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763375.901327:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763375.901337:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3350 00000100:00080000:3.0:1789763375.901351:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763375.901358:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1789763375.901366:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789763375.901374:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763375.901406:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699513045120/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763375.901427:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763375/real 1789763375] req@ffff8dc0107af800 x1876699513044864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763375.901430:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763375.901455:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763375.901461:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763375.901470:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699513044864/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763375.901477:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3350 00000100:00080000:3.0:1789763375.901496:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763375.901501:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763375.901508:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00080000:2.0:1789763375.901511:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763375.901524:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513044864:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789763375.901526:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763375.901549:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699513045248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763375.901561:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513044992:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763375.901592:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763375.901606:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763375/real 1789763375] req@ffff8dc0107ae680 x1876699513044992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763375.901616:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699513044992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763375.901623:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763375.901626:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763375.901629:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513044992:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763375.901635:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513045120:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789763375.901649:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763375.901656:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763375/real 1789763375] req@ffff8dc0107ace00 x1876699513045120/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763375.901664:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699513045120/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763375.901670:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763375.901671:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763375.901673:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513045120:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763375.901688:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513045248:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763375.901710:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763375.901723:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763375/real 1789763375] req@ffff8dc0107af100 x1876699513045248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763375.901735:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699513045248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763375.901744:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763375.901748:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763375.901752:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513045248:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763376.925118:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763377.949077:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763378.973144:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763379.997120:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763381.021122:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763381.021157:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763381.021167:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763381.021186:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3355 00000100:00080000:2.0:1789763381.021206:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:0.0:1789763381.021218:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763381.021222:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763381.021321:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699513045376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763381.021368:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763381.021375:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763381.021378:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763381.021386:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3355 00000100:00080000:2.0:1789763381.021393:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763381.021400:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763381.021416:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763381.021434:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513045376:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763381.021436:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699513045504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763381.021465:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763381.021470:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763381.021473:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763381.021479:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3355 00000100:00080000:2.0:1789763381.021484:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763381.021489:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763381.021496:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763381.021510:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699513045632/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763381.021525:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763381.021529:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763381.021532:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763381.021537:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3355 00000100:00080000:2.0:1789763381.021542:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763381.021547:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763381.021554:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763381.021559:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763381.021568:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699513045760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763381.021626:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763381/real 1789763381] req@ffff8dc00368c700 x1876699513045376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763381.021659:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699513045376/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763381.021678:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763381.021685:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763381.021692:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513045376:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763381.021732:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513045504:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763381.021809:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763381.021833:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763381/real 1789763381] req@ffff8dc00368d180 x1876699513045504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763381.021859:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699513045504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763381.021876:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763381.021883:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763381.021891:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513045504:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763381.021904:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513045632:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763381.021980:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763381.021997:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763381/real 1789763381] req@ffff8dc00368e680 x1876699513045632/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763381.022023:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699513045632/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763381.022034:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763381.022038:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763381.022042:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513045632:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763381.022050:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513045760:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763381.022088:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763381.022098:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763381/real 1789763381] req@ffff8dc00368dc00 x1876699513045760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763381.022110:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699513045760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763381.022118:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763381.022136:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763381.022140:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513045760:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763382.045186:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763383.069149:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763384.093089:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763385.117084:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763386.141100:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763386.141113:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763386.141119:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789763386.141133:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763386.141147:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3360 00000100:00080000:3.0:1789763386.141165:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763386.141179:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763386.141253:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699513045888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763386.141302:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763386.141310:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763386.141314:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763386.141322:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3360 00000100:00080000:3.0:1789763386.141332:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763386.141340:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513045888:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763386.141348:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763386.141367:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763386.141397:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699513046016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763386.141413:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763386.141418:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763386.141422:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763386.141427:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3360 00000100:00080000:3.0:1789763386.141434:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789763386.141444:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763386.141446:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763386.141532:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763386/real 1789763386] req@ffff8dc0107aea00 x1876699513045888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789763386.141562:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763386.141571:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699513045888/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763386.141596:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789763386.141601:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699513046144/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763386.141608:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763386.141624:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513045888:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763386.141657:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513046016:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763386.141695:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763386.141741:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763386/real 1789763386] req@ffff8dc0107ad880 x1876699513046016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763386.141761:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699513046016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763386.141776:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763386.141780:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763386.141785:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513046016:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1789763386.141880:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763386.141887:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763386.141891:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1789763386.141953:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513046144:192.168.203.156@tcp:8:kworker.0 00000100:00080000:3.0:1789763386.142026:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3360 00000800:00000100:2.0:1789763386.142028:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763386.142042:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763386.142055:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763386.142067:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763386.142071:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763386/real 1789763386] req@ffff8dc0107af100 x1876699513046144/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763386.142110:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699513046272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763386.142121:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699513046144/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763386.142151:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763386.142160:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763386.142167:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513046144:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763386.142207:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513046272:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763386.142238:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763386.142254:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763386/real 1789763386] req@ffff8dc0107ace00 x1876699513046272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763386.142270:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699513046272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763386.142281:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763386.142285:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763386.142289:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513046272:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763387.165079:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763388.189143:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763389.213102:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763390.237074:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763391.261047:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763391.261065:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763391.261073:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763391.261093:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3365 00000100:00080000:3.0:1789763391.261108:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763391.261118:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789763391.261153:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763391.261169:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699513046400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763391.261220:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763391.261228:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763391.261231:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763391.261241:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3365 00000100:00080000:3.0:1789763391.261251:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763391.261261:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763391.261278:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763391.261287:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513046400:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763391.261308:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699513046528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763391.261327:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763391.261331:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763391.261334:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763391.261340:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3365 00000100:00080000:3.0:1789763391.261345:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763391.261350:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763391.261357:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763391.261372:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699513046656/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763391.261397:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789763391.261408:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763391.261409:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763391.261416:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763391.261426:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3365 00000100:00080000:3.0:1789763391.261433:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763391.261439:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763391.261447:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763391.261462:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699513046784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763391.261504:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763391/real 1789763391] req@ffff8dc0107ae680 x1876699513046400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763391.261541:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699513046400/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763391.261561:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763391.261569:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763391.261577:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513046400:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763391.261617:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513046528:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763391.261693:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763391.261733:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763391/real 1789763391] req@ffff8dc0107af800 x1876699513046528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763391.261762:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699513046528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763391.261779:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763391.261787:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763391.261794:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513046528:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763391.261809:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513046656:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789763391.261850:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763391.261873:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763391/real 1789763391] req@ffff8dc0107ac380 x1876699513046656/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763391.261941:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699513046656/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763391.261961:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763391.261969:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763391.261978:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513046656:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763391.261992:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513046784:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763391.262034:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763391.262056:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763391/real 1789763391] req@ffff8dc0107adf80 x1876699513046784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763391.262091:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699513046784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763391.262107:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763391.262113:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763391.262119:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513046784:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763392.285079:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763393.309129:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763394.333231:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763395.357085:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763396.381040:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763396.381056:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763396.381062:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763396.381078:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3370 00000100:00080000:3.0:1789763396.381095:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763396.381105:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789763396.381115:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763396.381171:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699513046912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763396.381212:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763396.381218:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763396.381222:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763396.381247:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3370 00000100:00080000:3.0:1789763396.381254:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763396.381263:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763396.381272:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513046912:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789763396.381279:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763396.381305:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699513047040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763396.381319:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763396.381325:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763396.381327:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763396.381331:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3370 00000100:00080000:3.0:1789763396.381339:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763396.381343:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763396.381350:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:2.0:1789763396.381351:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763396.381369:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699513047168/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763396.381383:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763396.381405:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763396.381408:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763396.381414:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3370 00000100:00080000:3.0:1789763396.381419:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763396.381424:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763396.381432:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763396.381440:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763396/real 1789763396] req@ffff8dc0107aed80 x1876699513046912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763396.381452:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699513047296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763396.381474:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699513046912/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763396.381491:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763396.381497:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763396.381503:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513046912:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763396.381533:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513047040:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763396.381561:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763396.381577:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763396/real 1789763396] req@ffff8dc0107ac700 x1876699513047040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763396.381591:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699513047040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763396.381602:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763396.381606:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763396.381610:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513047040:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763396.381616:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513047168:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789763396.381684:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763396.381698:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763396/real 1789763396] req@ffff8dc0107ad180 x1876699513047168/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763396.381711:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699513047168/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763396.381721:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763396.381724:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763396.381728:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513047168:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763396.381734:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513047296:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763396.381752:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763396.381763:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763396/real 1789763396] req@ffff8dc0107af480 x1876699513047296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763396.381775:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699513047296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763396.381784:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763396.381786:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763396.381790:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513047296:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763397.405120:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763398.429159:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763399.453179:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763400.477171:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763401.501082:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763401.501094:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763401.501100:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763401.501114:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3376 00000100:00080000:1.0:1789763401.501127:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763401.501137:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789763401.501169:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763401.501189:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699513047424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763401.501228:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763401.501233:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763401.501235:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763401.501240:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3376 00000100:00080000:1.0:1789763401.501246:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763401.501251:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763401.501263:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763401.501289:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513047424:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763401.501290:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699513047552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763401.501317:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763401.501320:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763401.501323:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763401.501329:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3376 00000100:00080000:1.0:1789763401.501333:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763401.501337:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763401.501345:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763401.501361:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699513047680/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763401.501383:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763401.501387:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763401.501390:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763401.501394:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3376 00000100:00080000:1.0:1789763401.501399:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789763401.501407:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789763401.501413:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763401.501422:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763401.501438:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699513047808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763401.501478:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763401/real 1789763401] req@ffff8dc005c94380 x1876699513047424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763401.501513:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699513047424/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763401.501530:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763401.501535:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763401.501540:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513047424:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763401.501571:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513047552:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763401.501629:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763401.501649:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763401/real 1789763401] req@ffff8dc005c97b80 x1876699513047552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763401.501665:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699513047552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763401.501677:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763401.501681:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763401.501686:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513047552:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763401.501697:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513047680:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789763401.501721:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763401.501733:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763401/real 1789763401] req@ffff8dc005c96300 x1876699513047680/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763401.501747:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699513047680/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763401.501758:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763401.501761:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763401.501765:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513047680:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763401.501772:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513047808:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763401.501791:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763401.501802:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763401/real 1789763401] req@ffff8dc005c95500 x1876699513047808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763401.501816:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699513047808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763401.501826:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763401.501829:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763401.501833:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513047808:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763402.525084:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763403.549036:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763404.573026:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763405.597211:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763406.621159:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763406.621191:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763406.621197:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763406.621214:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3381 00000100:00080000:2.0:1789763406.621231:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763406.621242:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763406.621306:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699513047936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763406.621354:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763406.621359:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763406.621361:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763406.621367:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3381 00000100:00080000:2.0:1789763406.621373:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763406.621378:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763406.621396:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00010000:00010000:3.0:1789763406.621418:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763406.621421:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699513048064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763406.621436:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513047936:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763406.621465:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763406.621471:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763406.621473:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763406.621478:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3381 00000100:00080000:2.0:1789763406.621482:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763406.621486:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763406.621494:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763406.621514:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699513048192/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763406.621530:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763406.621536:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763406.621540:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763406.621545:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3381 00000100:00080000:2.0:1789763406.621594:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763406.621600:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763406.621607:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763406.621624:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763406.621625:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699513048320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763406.621726:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763406/real 1789763406] req@ffff8dc00368df80 x1876699513047936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763406.621779:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699513047936/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763406.621801:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763406.621809:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763406.621818:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513047936:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763406.621863:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513048064:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763406.621942:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763406.621963:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763406/real 1789763406] req@ffff8dc00368f800 x1876699513048064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763406.621980:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699513048064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763406.621992:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763406.621996:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763406.622011:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513048064:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763406.622020:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513048192:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763406.622057:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763406.622082:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763406/real 1789763406] req@ffff8dc00368e300 x1876699513048192/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763406.622107:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699513048192/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763406.622140:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763406.622145:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763406.622150:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513048192:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763406.622160:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513048320:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763406.622193:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763406.622209:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763406/real 1789763406] req@ffff8dc00368ed80 x1876699513048320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763406.622223:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699513048320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763406.622233:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763406.622237:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763406.622241:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513048320:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763407.645118:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763408.669153:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763409.693224:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763410.717101:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763411.741059:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763411.741069:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763411.741074:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763411.741084:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3386 00010000:00010000:2.0:1789763411.741087:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763411.741106:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763411.741115:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763411.741149:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699513048448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763411.741171:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763411.741174:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763411.741175:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763411.741178:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3386 00000100:00080000:1.0:1789763411.741181:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763411.741184:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763411.741197:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763411.741205:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699513048576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763411.741212:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763411.741215:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763411.741216:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763411.741220:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3386 00000100:00080000:1.0:1789763411.741223:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763411.741226:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763411.741229:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763411.741236:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699513048704/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763411.741245:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763411.741249:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763411.741253:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763411.741255:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513048448:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789763411.741262:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3386 00000100:00080000:1.0:1789763411.741271:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763411.741275:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763411.741278:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763411.741291:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699513048832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763411.741415:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513048576:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763411.741474:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513048704:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763411.741503:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513048832:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763412.765153:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763413.789112:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763414.813372:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763414.813386:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:2.0:1789763414.813487:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763414.813556:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789763414.813615:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789763414.813629:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789763414.813640:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:0.0:1789763414.813712:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763411/real 1789763414] req@ffff8dc005c96680 x1876699513048704/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763414.813765:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699513048704/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763414.813793:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763414.813804:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763414.813812:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513048704:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763414.813843:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763411/real 1789763414] req@ffff8dc005c95880 x1876699513048832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763414.813877:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699513048832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763414.813893:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763414.813948:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763414.813956:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513048832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763414.813976:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763411/real 1789763414] req@ffff8dc005c94e00 x1876699513048448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763414.814003:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699513048448/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763414.814028:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763414.814037:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763414.814043:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513048448:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763414.814067:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763411/real 1789763414] req@ffff8dc005c97480 x1876699513048576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763436 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763414.814089:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699513048576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763436 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763414.814106:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763414.814112:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763414.814118:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513048576:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763415.837233:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763416.861142:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763416.861165:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763416.861175:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763416.861198:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3391 00010000:00010000:3.0:1789763416.861200:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763416.861215:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763416.861231:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763416.861337:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699513048960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763416.861420:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763416.861427:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763416.861429:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763416.861437:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3391 00000100:00080000:2.0:1789763416.861444:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763416.861451:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763416.861468:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763416.861511:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699513049088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763416.861529:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513048960:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763416.861538:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763416.861551:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763416.861554:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763416.861562:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3391 00000100:00080000:2.0:1789763416.861569:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763416.861577:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763416.861586:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763416.861610:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699513049216/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763416.861626:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763416.861632:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763416.861636:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763416.861646:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3391 00000100:00080000:2.0:1789763416.861662:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763416.861668:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763416.861675:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763416.861680:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763416.861703:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699513049344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763416.861779:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763416/real 1789763416] req@ffff8dc00368ca80 x1876699513048960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763416.861816:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699513048960/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763416.861838:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763416.861847:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763416.861855:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513048960:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763416.861951:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513049088:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763416.862009:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763416.862038:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763416/real 1789763416] req@ffff8dc00368ea00 x1876699513049088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763416.862064:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699513049088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763416.862081:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763416.862089:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763416.862096:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513049088:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763416.862112:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513049216:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763416.862160:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763416.862186:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763416/real 1789763416] req@ffff8dc00368c000 x1876699513049216/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763416.862214:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699513049216/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763416.862232:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763416.862240:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763416.862246:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513049216:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763416.862260:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513049344:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763416.862305:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763416.862330:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763416/real 1789763416] req@ffff8dc00368ce00 x1876699513049344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763416.862355:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699513049344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763416.862374:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763416.862381:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763416.862388:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513049344:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763417.885155:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763418.909135:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763419.933146:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763420.957177:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763421.981155:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763421.981169:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763421.981182:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763421.981190:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763421.981205:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3396 00000100:00080000:1.0:1789763421.981219:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763421.981228:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763421.981279:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96d80 x1876699513049472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763421.981327:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763421.981334:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763421.981337:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763421.981345:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3396 00000100:00080000:1.0:1789763421.981353:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763421.981360:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763421.981375:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763421.981400:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513049472:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763421.981405:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699513049600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763421.981461:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763421.981469:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763421.981475:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763421.981485:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3396 00000100:00080000:1.0:1789763421.981495:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763421.981505:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763421.981516:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763421.981540:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699513049728/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763421.981554:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763421.981558:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763421.981561:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763421.981567:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3396 00000100:00080000:1.0:1789763421.981572:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763421.981580:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763421.981587:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763421.981599:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699513049856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763421.982089:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763421.982182:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763421/real 1789763421] req@ffff8dc005c96d80 x1876699513049472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763421.982212:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96d80 x1876699513049472/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763421.982230:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763421.982236:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763421.982242:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513049472:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763421.982282:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513049600:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763421.982320:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763421.982338:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763421/real 1789763421] req@ffff8dc005c97480 x1876699513049600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763421.982355:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699513049600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763421.982366:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763421.982371:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763421.982375:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513049600:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789763421.982383:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513049728:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789763421.982411:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763421.982424:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763421/real 1789763421] req@ffff8dc005c94e00 x1876699513049728/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763421.982468:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699513049728/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763421.982480:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763421.982485:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763421.982489:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513049728:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789763421.982498:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513049856:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763421.982529:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763421.982544:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763421/real 1789763421] req@ffff8dc005c95880 x1876699513049856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763421.982559:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699513049856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763421.982569:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763421.982574:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763421.982577:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513049856:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763423.005140:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763424.029203:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763425.053166:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763426.077203:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763427.101107:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763427.101121:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763427.101127:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763427.101141:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3401 00000100:00080000:1.0:1789763427.101160:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763427.101170:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789763427.101198:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763427.101224:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699513049984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763427.101262:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763427.101267:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763427.101270:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763427.101275:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3401 00000100:00080000:1.0:1789763427.101283:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763427.101289:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763427.101304:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763427.101321:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699513050112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763427.101343:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513049984:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789763427.101344:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763427.101366:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763427.101376:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763427.101385:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3401 00000100:00080000:1.0:1789763427.101389:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763427.101393:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763427.101425:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1789763427.101657:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763427.101717:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763427/real 1789763427] req@ffff8dc005c96680 x1876699513049984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763427.101741:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699513049984/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763427.101755:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763427.101760:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763427.101766:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513049984:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763427.101802:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513050112:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763427.101834:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763427.101849:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763427/real 1789763427] req@ffff8dc005c95500 x1876699513050112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763427.101865:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699513050112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763427.101875:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763427.101878:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763427.101882:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513050112:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763427.103202:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699513050240/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763427.103237:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763427.103242:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763427.103244:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763427.103250:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3401 00000100:00080000:1.0:1789763427.103256:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763427.103262:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763427.103269:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763427.103286:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699513050368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763427.103362:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513050240:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789763427.103441:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763427.103466:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763427/real 1789763427] req@ffff8dc005c96300 x1876699513050240/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763427.103496:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699513050240/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763427.103508:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763427.103513:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763427.103518:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513050240:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789763427.103525:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513050368:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763427.103549:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763427.103562:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763427/real 1789763427] req@ffff8dc005c97b80 x1876699513050368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763427.103576:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699513050368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763427.103586:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763427.103590:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763427.103594:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513050368:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763428.125162:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763429.149177:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763430.173177:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763431.198940:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763432.221213:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763432.223025:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763432.223046:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763432.223055:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763432.223073:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3406 00000100:00080000:3.0:1789763432.223091:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763432.223103:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763432.223170:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699513050496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763432.223213:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763432.223219:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763432.223222:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763432.223227:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3406 00000100:00080000:3.0:1789763432.223233:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763432.223240:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763432.223258:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763432.223279:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699513050624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763432.223291:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513050496:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763432.223314:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763432.223322:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763432.223324:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763432.223334:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3406 00000100:00080000:3.0:1789763432.223342:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763432.223348:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763432.223355:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763432.223398:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699513050752/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763432.223416:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763432.223420:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763432.223424:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763432.223439:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3406 00000800:00000100:0.0:1789763432.223441:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763432.223459:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763432.223473:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763432.223483:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763432.223503:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699513050880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763432.223543:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763432/real 1789763432] req@ffff8dc0107ad500 x1876699513050496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763432.223576:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699513050496/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763432.223593:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763432.223600:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763432.223610:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513050496:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763432.223677:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513050624:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763432.223708:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763432.223726:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763432/real 1789763432] req@ffff8dc0107ac000 x1876699513050624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763432.223743:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699513050624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763432.223754:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763432.223758:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763432.223762:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513050624:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763432.223769:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513050752:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763432.223791:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763432.223804:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763432/real 1789763432] req@ffff8dc0107afb80 x1876699513050752/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763432.223818:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699513050752/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763432.223836:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763432.223840:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763432.223844:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513050752:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763432.223851:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513050880:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763432.223870:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763432.223882:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763432/real 1789763432] req@ffff8dc0107ae300 x1876699513050880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763432.223895:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699513050880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763432.223940:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763432.223945:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763432.223949:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513050880:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763433.245087:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763434.269151:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763435.293087:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763436.317042:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763437.341047:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763437.341067:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763437.341075:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763437.341093:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3411 00000100:00080000:3.0:1789763437.341113:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1789763437.341129:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763437.341130:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763437.341191:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699513051008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763437.341235:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763437.341240:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763437.341243:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763437.341249:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3411 00000100:00080000:3.0:1789763437.341256:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763437.341265:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763437.341278:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763437.341298:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699513051136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763437.341318:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763437.341342:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513051008:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763437.341353:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763437.341359:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763437.341367:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3411 00000100:00080000:3.0:1789763437.341373:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763437.341381:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763437.341395:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763437.341418:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699513051264/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763437.341435:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763437.341440:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763437.341442:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763437.341448:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3411 00000100:00080000:3.0:1789763437.341458:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763437.341466:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789763437.341481:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789763437.341489:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763437.341527:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699513051392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763437.341554:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763437/real 1789763437] req@ffff8dc0107adc00 x1876699513051008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763437.341587:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699513051008/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763437.341604:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763437.341611:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763437.341616:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513051008:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763437.341655:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513051136:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763437.341684:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763437.341700:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763437/real 1789763437] req@ffff8dc0107aca80 x1876699513051136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763437.341716:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699513051136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763437.341727:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763437.341731:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763437.341736:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513051136:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763437.341744:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513051264:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763437.341766:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763437.341779:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763437/real 1789763437] req@ffff8dc0107ae300 x1876699513051264/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763437.341793:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699513051264/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763437.341803:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763437.341807:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763437.341811:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513051264:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763437.341818:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513051392:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763437.341838:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763437.341850:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763437/real 1789763437] req@ffff8dc0107afb80 x1876699513051392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763437.341865:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699513051392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763437.341875:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763437.341879:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763437.341882:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513051392:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763438.365209:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763439.389148:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763440.413198:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763441.437161:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763442.461118:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763442.461134:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763442.461140:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763442.461157:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3417 00000100:00080000:3.0:1789763442.461172:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763442.461184:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789763442.461211:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763442.461247:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699513051520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763442.461329:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763442.461336:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763442.461339:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763442.461346:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3417 00000100:00080000:3.0:1789763442.461351:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763442.461357:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763442.461370:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763442.461386:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513051520:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763442.461391:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699513051648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763442.461407:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763442.461411:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763442.461413:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763442.461418:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3417 00000100:00080000:3.0:1789763442.461422:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763442.461425:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763442.461431:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763442.461473:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699513051776/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763442.461487:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763442.461491:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763442.461493:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763442.461498:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3417 00000100:00080000:3.0:1789763442.461514:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763442.461519:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789763442.461527:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789763442.461529:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763442.461578:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699513051904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763442.461657:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763442/real 1789763442] req@ffff8dc0107ac000 x1876699513051520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763442.461721:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699513051520/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763442.461734:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763442.461739:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763442.461744:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513051520:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763442.461775:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513051648:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763442.461801:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763442.461816:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763442/real 1789763442] req@ffff8dc0107ad500 x1876699513051648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763442.461847:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699513051648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763442.461865:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763442.461869:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763442.461873:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513051648:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763442.461880:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513051776:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763442.461901:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763442.461955:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763442/real 1789763442] req@ffff8dc0107af480 x1876699513051776/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763442.461969:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699513051776/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763442.461990:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763442.461993:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763442.461996:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513051776:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763442.462002:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513051904:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763442.462022:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763442.462033:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763442/real 1789763442] req@ffff8dc0107ad180 x1876699513051904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763442.462044:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699513051904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763442.462052:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763442.462055:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763442.462059:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513051904:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763443.485153:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763444.509137:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763445.533193:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763446.557150:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763447.581025:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763447.581038:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763447.581044:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763447.581059:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3422 00000100:00080000:1.0:1789763447.581073:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763447.581083:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789763447.581088:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763447.581142:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699513052032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763447.581205:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763447.581213:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763447.581217:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763447.581227:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3422 00000100:00080000:1.0:1789763447.581236:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763447.581246:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763447.581288:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513052032:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1789763447.581307:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763447.581330:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699513052160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763447.581342:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763447.581346:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763447.581348:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763447.581353:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3422 00000100:00080000:1.0:1789763447.581358:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763447.581362:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763447.581368:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763447.581379:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699513052288/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763447.581389:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763447.581392:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763447.581394:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763447.581397:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3422 00000100:00080000:1.0:1789763447.581401:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789763447.581405:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789763447.581410:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763447.581415:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763447.581425:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699513052416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763447.581466:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763447/real 1789763447] req@ffff8dc005c94380 x1876699513052032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763447.581486:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699513052032/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763447.581497:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763447.581501:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763447.581506:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513052032:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763447.581534:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513052160:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763447.581553:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763447.581564:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763447/real 1789763447] req@ffff8dc005c95f80 x1876699513052160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763447.581575:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699513052160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763447.581583:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763447.581585:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763447.581589:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513052160:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763447.581593:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513052288:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763447.581608:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763447.581617:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763447/real 1789763447] req@ffff8dc005c94700 x1876699513052288/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763447.581627:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699513052288/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763447.581635:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763447.581637:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763447.581639:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513052288:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763447.581643:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513052416:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763447.581658:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763447.581666:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763447/real 1789763447] req@ffff8dc005c96a00 x1876699513052416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763447.581676:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699513052416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763447.581683:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763447.581685:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763447.581687:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513052416:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763448.605175:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763449.629218:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763450.653133:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763451.677098:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763452.701079:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763452.701097:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763452.701104:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1789763452.701114:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763452.701123:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3427 00000100:00080000:2.0:1789763452.701141:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763452.701153:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763452.701241:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699513052544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763452.701282:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763452.701288:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763452.701291:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763452.701297:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3427 00000100:00080000:2.0:1789763452.701303:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763452.701309:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763452.701326:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763452.701354:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513052544:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763452.701354:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699513052672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763452.701392:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763452.701397:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763452.701400:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763452.701406:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3427 00000100:00080000:2.0:1789763452.701412:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763452.701417:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763452.701424:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763452.701441:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699513052800/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763452.701457:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763452.701462:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763452.701470:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789763452.701471:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789763452.701486:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3427 00000100:00080000:2.0:1789763452.701493:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763452.701500:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763452.701508:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763452.701525:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699513052928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763452.701557:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763452/real 1789763452] req@ffff8dc00368f100 x1876699513052544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763452.701601:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699513052544/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763452.701627:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763452.701633:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763452.701639:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513052544:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763452.701679:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513052672:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763452.701728:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763452.701757:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763452/real 1789763452] req@ffff8dc00368f480 x1876699513052672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763452.701785:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699513052672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763452.701803:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763452.701810:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763452.701817:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513052672:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763452.701832:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513052800:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763452.701880:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763452.701937:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763452/real 1789763452] req@ffff8dc00368fb80 x1876699513052800/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763452.701955:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699513052800/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763452.701966:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763452.701971:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763452.701975:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513052800:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763452.701984:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513052928:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763452.702013:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763452.702028:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763452/real 1789763452] req@ffff8dc00368d880 x1876699513052928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763452.702043:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699513052928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763452.702053:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763452.702057:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763452.702062:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513052928:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763453.725222:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763454.749159:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763455.773157:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763456.797162:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763457.821223:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763457.821235:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763457.821240:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763457.821251:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3432 00000100:00080000:1.0:1789763457.821262:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763457.821270:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763457.821317:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699513053056/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1789763457.821359:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763457.821375:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763457.821383:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763457.821387:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763457.821394:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3432 00000100:00080000:1.0:1789763457.821402:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763457.821410:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763457.821422:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513053056:192.168.203.156@tcp:250:kworker.0 00000008:00100000:1.0:1789763457.821422:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763457.821484:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699513053184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763457.821514:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763457.821538:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763457.821541:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763457.821550:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3432 00000100:00080000:1.0:1789763457.821556:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789763457.821558:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789763457.821564:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763457.821574:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763457.821604:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699513053312/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763457.821635:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763457.821642:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763457.821645:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763457.821649:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763457/real 1789763457] req@ffff8dc005c95c00 x1876699513053056/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763457.821652:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3432 00000100:00080000:1.0:1789763457.821658:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763457.821665:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763457.821673:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763457.821685:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699513053056/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763457.821692:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699513053440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763457.821709:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763457.821717:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763457.821728:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513053056:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763457.821788:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513053184:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763457.821835:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763457.821877:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763457/real 1789763457] req@ffff8dc005c94a80 x1876699513053184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763457.821891:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699513053184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763457.821900:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763457.821903:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763457.821907:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513053184:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763457.821912:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513053312:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763457.821987:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763457.822023:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763457/real 1789763457] req@ffff8dc005c97100 x1876699513053312/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763457.822045:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699513053312/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763457.822075:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763457.822082:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763457.822090:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513053312:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763457.822102:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513053440:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763457.822168:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763457.822205:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763457/real 1789763457] req@ffff8dc005c97800 x1876699513053440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763457.822234:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699513053440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763457.822255:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763457.822277:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763457.822285:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513053440:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763458.845160:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763459.869225:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763460.893204:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763461.917241:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763462.941076:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763462.941090:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763462.941097:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763462.941111:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3437 00000100:00080000:1.0:1789763462.941133:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763462.941176:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789763462.941176:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763462.941235:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699513053568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763462.941272:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763462.941278:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763462.941280:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763462.941285:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3437 00000100:00080000:1.0:1789763462.941291:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763462.941298:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763462.941312:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763462.941332:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699513053696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763462.941349:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513053568:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789763462.941358:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763462.941371:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763462.941374:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763462.941381:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3437 00000100:00080000:1.0:1789763462.941388:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763462.941395:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763462.941401:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763462.941415:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699513053824/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763462.941428:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763462.941432:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763462.941434:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763462.941439:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3437 00000100:00080000:1.0:1789763462.941464:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763462.941471:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763462.941478:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763462.941490:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763462.941497:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699513053952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763462.941578:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763462/real 1789763462] req@ffff8dc005c95180 x1876699513053568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763462.941608:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699513053568/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763462.941626:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763462.941632:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763462.941637:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513053568:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763462.941676:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513053696:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763462.941706:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763462.941723:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763462/real 1789763462] req@ffff8dc005c97800 x1876699513053696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763462.941738:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699513053696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763462.941750:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763462.941754:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763462.941759:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513053696:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763462.941780:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513053824:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763462.941805:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763462.941818:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763462/real 1789763462] req@ffff8dc005c97100 x1876699513053824/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763462.941834:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699513053824/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763462.941845:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763462.941849:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763462.941853:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513053824:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763462.941860:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513053952:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763462.941881:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763462.941893:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763462/real 1789763462] req@ffff8dc005c94a80 x1876699513053952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763462.941938:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699513053952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763462.941950:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763462.941954:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763462.941958:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513053952:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763463.965198:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763464.989063:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763466.013190:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763467.037252:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763468.060988:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763468.060998:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763468.061004:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763468.061016:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3442 00000100:00080000:1.0:1789763468.061033:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763468.061040:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789763468.061083:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763468.061088:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699513054080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763468.061128:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763468.061131:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763468.061133:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763468.061136:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3442 00000100:00080000:1.0:1789763468.061140:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763468.061144:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763468.061155:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763468.061171:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513054080:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763468.061172:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699513054208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763468.061192:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763468.061195:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763468.061196:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763468.061201:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3442 00000100:00080000:1.0:1789763468.061204:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763468.061206:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763468.061211:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763468.061221:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699513054336/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763468.061230:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763468.061234:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763468.061235:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763468.061239:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3442 00000100:00080000:1.0:1789763468.061243:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763468.061247:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763468.061250:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763468.061252:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763468.061269:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699513054464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763468.061303:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763468/real 1789763468] req@ffff8dc005c95c00 x1876699513054080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763468.061319:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699513054080/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763468.061329:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763468.061332:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763468.061335:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513054080:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763468.061355:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513054208:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763468.061370:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763468.061379:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763468/real 1789763468] req@ffff8dc005c96a00 x1876699513054208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763468.061387:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699513054208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763468.061392:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763468.061394:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763468.061396:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513054208:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763468.061400:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513054336:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763468.061412:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763468.061418:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763468/real 1789763468] req@ffff8dc005c94700 x1876699513054336/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763468.061425:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699513054336/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763468.061434:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763468.061436:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763468.061438:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513054336:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763468.061442:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513054464:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763468.061453:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763468.061459:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763468/real 1789763468] req@ffff8dc005c95f80 x1876699513054464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763468.061466:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699513054464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763468.061471:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763468.061472:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763468.061474:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513054464:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763469.085194:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763470.109160:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763471.133018:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763472.157244:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763473.181080:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763473.181098:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763473.181106:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763473.181124:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3447 00000100:00080000:1.0:1789763473.181143:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763473.181157:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789763473.181176:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763473.181224:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699513054592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763473.181275:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763473.181283:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763473.181287:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763473.181298:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3447 00000100:00080000:1.0:1789763473.181307:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763473.181318:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763473.181346:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763473.181363:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513054592:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763473.181377:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699513054720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763473.181412:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763473.181417:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763473.181421:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763473.181426:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3447 00000100:00080000:1.0:1789763473.181432:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763473.181437:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763473.181446:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763473.181460:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699513054848/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763473.181475:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763473.181479:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763473.181481:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763473.181486:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3447 00000100:00080000:1.0:1789763473.181493:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763473.181498:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763473.181503:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763473.181521:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699513054976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763473.181528:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763473.181626:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763473/real 1789763473] req@ffff8dc005c94380 x1876699513054592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763473.181657:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699513054592/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763473.181687:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763473.181695:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763473.181702:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513054592:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763473.181743:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513054720:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763473.181780:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763473.181805:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763473/real 1789763473] req@ffff8dc005c97b80 x1876699513054720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763473.181832:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699513054720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763473.181852:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763473.181859:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763473.181867:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513054720:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763473.181881:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513054848:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763473.181955:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763473.181973:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763473/real 1789763473] req@ffff8dc005c96300 x1876699513054848/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763473.181989:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699513054848/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763473.182001:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763473.182005:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763473.182010:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513054848:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763473.182018:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513054976:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763473.182087:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763473.182107:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763473/real 1789763473] req@ffff8dc005c95500 x1876699513054976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763473.182129:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699513054976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763473.182142:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763473.182146:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763473.182150:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513054976:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763474.205211:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763475.229186:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763476.253254:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763477.277061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763478.301039:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763478.301051:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763478.301055:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763478.301068:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3452 00000100:00080000:1.0:1789763478.301080:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763478.301087:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789763478.301106:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763478.301131:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699513055104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763478.301157:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763478.301160:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763478.301161:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763478.301164:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3452 00000100:00080000:1.0:1789763478.301167:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763478.301170:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763478.301180:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763478.301193:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699513055232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763478.301194:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513055104:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789763478.301209:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763478.301211:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763478.301213:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763478.301215:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3452 00000100:00080000:1.0:1789763478.301217:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763478.301220:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763478.301224:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763478.301232:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699513055360/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763478.301239:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763478.301241:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763478.301242:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763478.301245:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3452 00000100:00080000:1.0:1789763478.301257:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763478.301272:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763478.301276:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763478.301284:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699513055488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763478.301324:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513055232:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763478.301352:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513055360:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763478.301366:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513055488:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763479.325174:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763480.349116:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763481.373367:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763481.373393:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:2.0:1789763481.373472:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763481.373495:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789763481.373547:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789763481.373556:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789763481.373567:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:0.0:1789763481.373648:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763478/real 1789763481] req@ffff8dc005c95880 x1876699513055232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763481.373689:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699513055232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763481.373711:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763481.373721:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763481.373728:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513055232:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763481.373743:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763478/real 1789763481] req@ffff8dc005c94e00 x1876699513055360/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763481.373765:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699513055360/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763481.373780:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763481.373786:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763481.373791:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513055360:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763481.373803:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763478/real 1789763481] req@ffff8dc005c97480 x1876699513055488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763481.373822:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699513055488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763481.373850:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763481.373857:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763481.373862:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513055488:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763481.373873:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763478/real 1789763481] req@ffff8dc005c96680 x1876699513055104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763481.373892:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699513055104/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763481.373940:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763481.373946:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763481.373951:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513055104:192.168.203.156@tcp:250:kworker.0 00010000:00010000:3.0:1789763482.397140:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763483.421055:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763483.421067:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763483.421072:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763483.421087:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3458 00000100:00080000:1.0:1789763483.421115:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763483.421124:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763483.421174:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96d80 x1876699513055616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763483.421233:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1789763483.421235:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763483.421242:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763483.421246:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763483.421253:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3458 00000100:00080000:1.0:1789763483.421261:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763483.421281:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763483.421295:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763483.421302:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513055616:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763483.421318:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699513055744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763483.421335:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763483.421340:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763483.421342:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763483.421347:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3458 00000100:00080000:1.0:1789763483.421353:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763483.421358:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763483.421364:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763483.421377:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699513055872/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763483.421386:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763483.421390:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763483.421392:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763483.421395:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3458 00000100:00080000:1.0:1789763483.421398:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763483.421401:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763483.421405:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1789763483.421412:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763483.421419:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699513056000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763483.421487:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763483/real 1789763483] req@ffff8dc005c96d80 x1876699513055616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763483.421512:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96d80 x1876699513055616/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763483.421525:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763483.421530:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763483.421535:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513055616:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763483.421569:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513055744:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763483.421600:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763483.421618:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763483/real 1789763483] req@ffff8dc005c96680 x1876699513055744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763483.421640:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699513055744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763483.421651:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763483.421656:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763483.421660:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513055744:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763483.421669:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513055872:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789763483.421696:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763483.421711:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763483/real 1789763483] req@ffff8dc005c97480 x1876699513055872/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763483.421737:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699513055872/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763483.421747:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763483.421750:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763483.421754:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513055872:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763483.421761:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513056000:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763483.421785:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763483.421798:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763483/real 1789763483] req@ffff8dc005c94e00 x1876699513056000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763483.421813:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699513056000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763483.421823:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763483.421827:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763483.421831:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513056000:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763484.445145:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763485.469168:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763486.493956:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763487.517139:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763488.541059:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763488.541072:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763488.541078:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763488.541093:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3463 00000100:00080000:1.0:1789763488.541107:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763488.541122:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789763488.541143:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763488.541188:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699513056128/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763488.541224:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763488.541229:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763488.541231:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763488.541236:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3463 00000100:00080000:1.0:1789763488.541242:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763488.541248:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763488.541261:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763488.541278:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513056128:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763488.541284:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699513056256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763488.541306:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763488.541311:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763488.541313:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763488.541317:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3463 00000100:00080000:1.0:1789763488.541322:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763488.541326:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763488.541332:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763488.541350:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699513056384/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763488.541364:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763488.541368:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763488.541371:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763488.541376:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3463 00000100:00080000:1.0:1789763488.541381:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763488.541391:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1789763488.541393:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1789763488.541401:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763488.541421:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699513056512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763488.541462:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763488/real 1789763488] req@ffff8dc005c95880 x1876699513056128/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763488.541488:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699513056128/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763488.541504:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763488.541509:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763488.541515:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513056128:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763488.541545:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513056256:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763488.541574:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763488.541589:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763488/real 1789763488] req@ffff8dc005c95500 x1876699513056256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763488.541605:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699513056256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763488.541615:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763488.541619:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763488.541624:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513056256:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763488.541633:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513056384:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789763488.541669:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763488.541690:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763488/real 1789763488] req@ffff8dc005c96300 x1876699513056384/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763488.541716:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699513056384/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763488.541733:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763488.541739:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763488.541743:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513056384:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763488.541752:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513056512:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763488.541798:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763488.541813:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763488/real 1789763488] req@ffff8dc005c97b80 x1876699513056512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763488.541836:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699513056512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763488.541847:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763488.541851:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763488.541855:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513056512:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763489.565128:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763490.589121:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763491.613134:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763492.637117:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763493.661134:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763493.661147:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763493.661153:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763493.661168:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3468 00010000:00010000:3.0:1789763493.661174:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763493.661184:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763493.661196:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763493.661256:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149f80 x1876699513056640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763493.661294:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763493.661299:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763493.661302:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763493.661308:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3468 00000100:00080000:0.0:1789763493.661313:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763493.661320:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763493.661334:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763493.661355:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514a300 x1876699513056768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763493.661375:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1789763493.661378:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513056640:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789763493.661381:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763493.661389:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763493.661404:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3468 00000100:00080000:0.0:1789763493.661409:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763493.661415:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763493.661422:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763493.661442:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148e00 x1876699513056896/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763493.661456:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763493.661463:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763493.661467:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763493.661474:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3468 00000100:00080000:0.0:1789763493.661480:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763493.661487:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763493.661494:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763493.661508:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763493.661516:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b100 x1876699513057024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763493.661706:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763493/real 1789763493] req@ffff8dc005149f80 x1876699513056640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763493.662050:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149f80 x1876699513056640/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763493.662157:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763493.662165:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763493.662172:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513056640:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763493.662204:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513056768:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763493.662243:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763493.662268:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763493/real 1789763493] req@ffff8dc00514a300 x1876699513056768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763493.662294:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514a300 x1876699513056768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763493.662311:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763493.662317:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763493.662324:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513056768:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789763493.662337:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513056896:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789763493.662378:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763493.662403:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763493/real 1789763493] req@ffff8dc005148e00 x1876699513056896/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763493.662442:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148e00 x1876699513056896/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763493.662459:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763493.662466:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763493.662473:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513056896:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789763493.662485:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513057024:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763493.662522:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763493.662539:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763493/real 1789763493] req@ffff8dc00514b100 x1876699513057024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763493.662554:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b100 x1876699513057024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763493.662565:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763493.662569:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763493.662572:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513057024:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763494.685217:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763495.709182:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763496.733132:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763497.757178:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763498.781117:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763498.781130:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763498.781138:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763498.781166:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3473 00010000:00010000:3.0:1789763498.781172:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763498.781181:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763498.781194:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763498.781257:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699513057152/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763498.781307:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763498.781312:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763498.781315:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763498.781321:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3473 00000100:00080000:2.0:1789763498.781328:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763498.781336:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763498.781348:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763498.781367:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513057152:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763498.781368:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699513057280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763498.781400:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763498.781407:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763498.781409:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763498.781415:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3473 00000100:00080000:2.0:1789763498.781418:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763498.781424:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763498.781432:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763498.781469:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699513057408/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763498.781489:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763498.781496:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763498.781500:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763498.781508:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3473 00000800:00000100:3.0:1789763498.781514:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789763498.781527:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763498.781542:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763498.781563:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763498.781588:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699513057536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763498.781596:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763498/real 1789763498] req@ffff8dc00368d880 x1876699513057152/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763498.781672:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699513057152/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763498.781684:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763498.781689:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763498.781693:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513057152:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763498.781725:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513057280:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763498.781745:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763498.781755:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763498/real 1789763498] req@ffff8dc00368fb80 x1876699513057280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763498.781764:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699513057280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763498.781771:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763498.781773:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763498.781776:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513057280:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789763498.781780:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513057408:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789763498.781816:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763498.781826:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763498/real 1789763498] req@ffff8dc00368f480 x1876699513057408/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763498.781839:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699513057408/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763498.781869:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763498.781887:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763498.781890:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513057408:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789763498.781896:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513057536:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763498.781935:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763498.781944:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763498/real 1789763498] req@ffff8dc00368f100 x1876699513057536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763498.781953:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699513057536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763498.781961:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763498.781965:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763498.781968:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513057536:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763499.805176:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763500.829296:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763501.853170:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763502.877127:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763503.901102:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763503.901119:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763503.901127:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763503.901145:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3478 00000100:00080000:1.0:1789763503.901182:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763503.901192:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789763503.901211:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763503.901256:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699513057664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763503.901295:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763503.901300:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763503.901303:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763503.901308:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3478 00000100:00080000:1.0:1789763503.901314:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763503.901319:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763503.901333:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763503.901345:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513057664:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763503.901353:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699513057792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763503.901369:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763503.901373:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763503.901375:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763503.901380:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3478 00000100:00080000:1.0:1789763503.901385:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763503.901389:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763503.901394:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763503.901408:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699513057920/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763503.901420:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763503.901424:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763503.901427:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763503.901431:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3478 00000100:00080000:1.0:1789763503.901435:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763503.901440:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763503.901445:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763503.901456:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699513058048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1789763503.901471:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763503.901572:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763503/real 1789763503] req@ffff8dc005c94380 x1876699513057664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763503.901601:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699513057664/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763503.901619:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763503.901624:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763503.901630:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513057664:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763503.901666:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513057792:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763503.901699:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763503.901716:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763503/real 1789763503] req@ffff8dc005c95f80 x1876699513057792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763503.901732:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699513057792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763503.901743:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763503.901748:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763503.901752:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513057792:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789763503.901759:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513057920:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789763503.901795:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763503.901808:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763503/real 1789763503] req@ffff8dc005c94700 x1876699513057920/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763503.901823:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699513057920/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763503.901877:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763503.901883:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763503.901888:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513057920:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789763503.901896:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513058048:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763503.901978:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763503.901996:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763503/real 1789763503] req@ffff8dc005c96a00 x1876699513058048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763503.902012:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699513058048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763503.902023:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763503.902027:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763503.902031:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513058048:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763504.925194:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763505.949180:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763506.973190:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763507.997122:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763509.021091:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763509.021105:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763509.021111:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763509.021125:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3483 00000100:00080000:2.0:1789763509.021139:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763509.021162:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789763509.021168:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763509.021225:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699513058176/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763509.021262:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763509.021267:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763509.021269:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763509.021275:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3483 00000100:00080000:2.0:1789763509.021280:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763509.021287:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763509.021300:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763509.021315:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513058176:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763509.021322:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699513058304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763509.021341:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763509.021344:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763509.021347:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763509.021352:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3483 00000100:00080000:2.0:1789763509.021356:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763509.021359:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763509.021365:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763509.021382:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699513058432/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763509.021397:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763509.021401:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763509.021404:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763509.021410:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3483 00000100:00080000:2.0:1789763509.021414:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763509.021420:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:3.0:1789763509.021428:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:2.0:1789763509.021432:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763509.021451:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699513058560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763509.021498:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763509/real 1789763509] req@ffff8dc00368ce00 x1876699513058176/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763509.021535:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699513058176/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763509.021550:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763509.021555:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763509.021561:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513058176:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763509.021590:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513058304:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763509.021616:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763509.021631:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763509/real 1789763509] req@ffff8dc00368c000 x1876699513058304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763509.021646:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699513058304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763509.021656:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763509.021660:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763509.021665:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513058304:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789763509.021671:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513058432:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789763509.021692:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763509.021704:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763509/real 1789763509] req@ffff8dc00368ea00 x1876699513058432/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763509.021718:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699513058432/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763509.021728:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763509.021731:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763509.021735:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513058432:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789763509.021742:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513058560:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763509.021772:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763509.021783:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763509/real 1789763509] req@ffff8dc00368ca80 x1876699513058560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763509.021820:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699513058560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763509.021830:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763509.021834:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763509.021838:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513058560:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763510.045218:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763511.069152:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763512.093184:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763513.117069:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763514.141045:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763514.141057:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763514.141080:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763514.141094:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3488 00000100:00080000:2.0:1789763514.141109:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763514.141119:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789763514.141168:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763514.141189:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699513058688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763514.141247:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763514.141255:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763514.141276:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763514.141287:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3488 00000100:00080000:2.0:1789763514.141297:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789763514.141306:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513058688:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763514.141317:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763514.141348:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763514.141375:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699513058816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763514.141397:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763514.141402:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763514.141404:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763514.141410:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3488 00000100:00080000:2.0:1789763514.141418:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763514.141424:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763514.141432:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1789763514.141436:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763514.141452:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699513058944/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763514.141472:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763514.141476:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763514.141479:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763514.141485:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3488 00000100:00080000:2.0:1789763514.141491:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763514.141498:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763514.141507:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763514.141516:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763514/real 1789763514] req@ffff8dc00368ed80 x1876699513058688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763514.141531:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699513059072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763514.141560:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699513058688/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763514.141578:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763514.141588:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763514.141595:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513058688:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763514.141634:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513058816:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763514.141672:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763514.141697:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763514/real 1789763514] req@ffff8dc00368e300 x1876699513058816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763514.141739:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699513058816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763514.141757:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763514.141790:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763514.141799:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513058816:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789763514.141813:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513058944:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789763514.141858:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763514.141881:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763514/real 1789763514] req@ffff8dc00368f800 x1876699513058944/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763514.141939:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699513058944/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763514.141951:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763514.141956:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763514.141961:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513058944:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789763514.141970:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513059072:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763514.142000:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763514.142015:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763514/real 1789763514] req@ffff8dc00368df80 x1876699513059072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763514.142029:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699513059072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763514.142048:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763514.142053:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763514.142057:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513059072:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763515.165163:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763516.189122:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763517.213200:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763518.237054:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763519.261070:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763519.261083:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763519.261091:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763519.261106:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3493 00000100:00080000:2.0:1789763519.261123:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763519.261136:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789763519.261175:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763519.261187:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699513059200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763519.261225:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763519.261230:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763519.261232:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763519.261238:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3493 00000100:00080000:2.0:1789763519.261243:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763519.261249:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763519.261261:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763519.261281:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699513059328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763519.261292:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513059200:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763519.261300:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763519.261308:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763519.261310:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763519.261316:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3493 00000100:00080000:2.0:1789763519.261320:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763519.261325:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763519.261332:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763519.261348:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699513059456/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763519.261372:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763519.261379:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763519.261383:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1789763519.261392:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789763519.261412:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3493 00000100:00080000:2.0:1789763519.261426:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763519.261435:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763519.261446:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763519.261453:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763519/real 1789763519] req@ffff8dc00368dc00 x1876699513059200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763519.261479:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699513059584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763519.261485:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699513059200/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763519.261545:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763519.261557:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763519.261568:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513059200:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763519.261622:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513059328:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763519.261668:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763519.261696:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763519/real 1789763519] req@ffff8dc00368e680 x1876699513059328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763519.261759:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699513059328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763519.261778:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763519.261783:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763519.261788:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513059328:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789763519.261797:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513059456:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789763519.261835:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763519.261859:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763519/real 1789763519] req@ffff8dc00368d180 x1876699513059456/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763519.261885:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699513059456/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763519.261968:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763519.261978:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763519.261985:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513059456:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789763519.261998:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513059584:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763519.262043:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763519.262067:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763519/real 1789763519] req@ffff8dc00368c700 x1876699513059584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763519.262084:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699513059584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763519.262094:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763519.262098:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763519.262103:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513059584:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763520.285070:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763521.309142:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763522.333250:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763523.357223:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763524.381150:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763524.381163:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763524.381168:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763524.381184:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3498 00010000:00010000:1.0:1789763524.381200:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763524.381205:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763524.381222:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763524.381275:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699513059712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763524.381325:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763524.381333:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763524.381337:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763524.381355:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3498 00000100:00080000:3.0:1789763524.381364:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763524.381373:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763524.381387:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763524.381390:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513059712:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763524.381411:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699513059840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763524.381429:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763524.381449:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763524.381455:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763524.381460:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3498 00000100:00080000:3.0:1789763524.381492:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763524.381498:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763524.381505:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:2.0:1789763524.381515:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763524.381531:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699513059968/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763524.381557:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763524.381562:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763524.381565:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763524.381569:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3498 00000100:00080000:3.0:1789763524.381574:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763524.381580:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763524.381588:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763524.381599:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763524/real 1789763524] req@ffff8dc0107ac700 x1876699513059712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763524.381608:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699513060096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763524.381662:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699513059712/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763524.381681:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763524.381727:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763524.381733:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513059712:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763524.381764:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513059840:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763524.381829:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763524.381866:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763524/real 1789763524] req@ffff8dc0107aed80 x1876699513059840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763524.381883:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699513059840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763524.381894:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763524.381898:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763524.381902:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513059840:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763524.381909:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513059968:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789763524.381966:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763524.381981:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763524/real 1789763524] req@ffff8dc0107adf80 x1876699513059968/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763524.381996:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699513059968/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763524.382005:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763524.382009:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763524.382013:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513059968:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763524.382019:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513060096:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763524.382039:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763524.382050:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763524/real 1789763524] req@ffff8dc0107ac380 x1876699513060096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763524.382063:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699513060096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763524.382094:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763524.382101:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763524.382107:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513060096:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763525.405130:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763526.429150:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763527.453147:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763528.477018:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763529.501055:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763529.501066:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:1.0:1789763529.501067:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763529.501073:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763529.501098:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3504 00000100:00080000:3.0:1789763529.501111:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763529.501119:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763529.501167:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699513060224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763529.501192:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763529.501196:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763529.501198:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763529.501203:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3504 00000100:00080000:3.0:1789763529.501206:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763529.501210:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763529.501222:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763529.501236:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699513060352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763529.501244:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513060224:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763529.501247:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763529.501252:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763529.501254:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763529.501258:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3504 00000100:00080000:3.0:1789763529.501261:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763529.501263:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763529.501268:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763529.501277:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699513060480/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763529.501284:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763529.501288:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763529.501289:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763529.501296:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3504 00000100:00080000:3.0:1789763529.501301:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763529.501307:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763529.501314:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763529.501324:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699513060608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763529.501334:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763529.501387:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763529/real 1789763529] req@ffff8dc0107af800 x1876699513060224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763529.501412:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699513060224/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763529.501423:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763529.501426:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763529.501430:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513060224:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763529.501457:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513060352:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763529.501476:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763529.501487:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763529/real 1789763529] req@ffff8dc0107ae680 x1876699513060352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763529.501496:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699513060352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763529.501503:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763529.501524:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763529.501529:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513060352:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763529.501537:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513060480:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763529.501566:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763529.501579:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763529/real 1789763529] req@ffff8dc0107ace00 x1876699513060480/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763529.501621:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699513060480/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763529.501640:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763529.501644:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763529.501687:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513060480:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763529.501710:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513060608:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763529.501728:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763529.501745:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763529/real 1789763529] req@ffff8dc0107af100 x1876699513060608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763529.501754:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699513060608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763529.501761:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763529.501764:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763529.501767:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513060608:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763530.525150:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763531.549175:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763532.573152:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763533.597196:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763534.621105:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763534.621137:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763534.621145:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763534.621162:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3509 00000100:00080000:3.0:1789763534.621181:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763534.621192:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763534.621205:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763534.621243:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699513060736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763534.621282:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763534.621287:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763534.621290:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763534.621295:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3509 00000100:00080000:3.0:1789763534.621300:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763534.621304:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763534.621317:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763534.621335:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699513060864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763534.621359:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513060736:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763534.621364:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763534.621374:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763534.621377:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763534.621383:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3509 00000100:00080000:3.0:1789763534.621387:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763534.621391:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763534.621398:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763534.621416:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699513060992/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763534.621431:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763534.621436:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763534.621439:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763534.621444:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3509 00000100:00080000:3.0:1789763534.621449:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763534.621455:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:1.0:1789763534.621465:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789763534.621468:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763534.621488:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699513061120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763534.621530:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763534/real 1789763534] req@ffff8dc0107ad880 x1876699513060736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763534.621558:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699513060736/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763534.621589:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763534.621594:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763534.621600:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513060736:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763534.621673:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513060864:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763534.621722:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763534.621750:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763534/real 1789763534] req@ffff8dc0107aea00 x1876699513060864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763534.621772:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699513060864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763534.621784:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763534.621788:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763534.621793:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513060864:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763534.621801:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513060992:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763534.621848:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763534.621863:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763534/real 1789763534] req@ffff8dc0107af100 x1876699513060992/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763534.621878:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699513060992/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763534.621889:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763534.621893:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763534.621897:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513060992:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763534.621957:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513061120:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763534.621983:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763534.621998:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763534/real 1789763534] req@ffff8dc0107ace00 x1876699513061120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763534.622013:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699513061120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763534.622023:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763534.622027:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763534.622031:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513061120:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763535.645208:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763536.669188:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763537.693135:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763538.717147:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763539.741006:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763539.741019:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:0.0:1789763539.741020:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763539.741024:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763539.741044:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3514 00000100:00080000:3.0:1789763539.741055:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763539.741062:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763539.741108:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699513061248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763539.741133:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763539.741137:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763539.741138:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763539.741142:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3514 00000100:00080000:3.0:1789763539.741147:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763539.741152:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763539.741171:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513061248:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789763539.741171:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763539.741191:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699513061376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763539.741202:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763539.741205:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763539.741206:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763539.741210:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3514 00000100:00080000:3.0:1789763539.741213:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763539.741217:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763539.741221:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763539.741233:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699513061504/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763539.741242:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763539.741245:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763539.741247:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763539.741250:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3514 00000100:00080000:3.0:1789763539.741253:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763539.741255:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763539.741259:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763539.741264:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763539.741272:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699513061632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763539.741306:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763539/real 1789763539] req@ffff8dc0107ae680 x1876699513061248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763539.741323:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699513061248/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763539.741333:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763539.741336:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763539.741340:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513061248:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763539.741366:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513061376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763539.741381:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763539.741390:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763539/real 1789763539] req@ffff8dc0107af800 x1876699513061376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763539.741399:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699513061376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763539.741404:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763539.741407:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763539.741409:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513061376:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763539.741412:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513061504:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763539.741424:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763539.741430:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763539/real 1789763539] req@ffff8dc0107ac380 x1876699513061504/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763539.741437:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699513061504/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763539.741444:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763539.741445:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763539.741447:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513061504:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763539.741451:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513061632:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763539.741462:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763539.741468:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763539/real 1789763539] req@ffff8dc0107adf80 x1876699513061632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763564 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763539.741480:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699513061632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763564 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763539.741486:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763539.741487:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763539.741489:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513061632:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763540.765170:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763541.789201:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763542.813081:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763543.837232:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763544.861175:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763544.861189:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763544.861206:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763544.861215:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763544.861235:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3519 00000100:00080000:1.0:1789763544.861253:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763544.861267:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763544.861341:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699513061760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763544.861396:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763544.861401:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763544.861404:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763544.861409:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3519 00000100:00080000:1.0:1789763544.861415:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763544.861419:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763544.861432:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763544.861451:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699513061888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763544.861466:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763544.861471:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763544.861473:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763544.861477:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3519 00000100:00080000:1.0:1789763544.861491:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763544.861495:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763544.861502:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763544.861508:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513061760:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763544.861521:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699513062016/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763544.861570:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763544.861577:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763544.861581:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763544.861587:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3519 00000100:00080000:1.0:1789763544.861593:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763544.861599:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763544.861607:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763544.861636:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699513062144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763544.861737:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513061888:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763544.861797:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513062016:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763544.861830:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513062144:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763545.885223:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763546.909264:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789763547.933326:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00010000:00010000:3.0:1789763547.933360:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1789763547.933419:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1789763547.933442:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789763547.933502:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789763547.933563:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:1.0:1789763547.933575:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1789763547.933586:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763544/real 1789763547] req@ffff8dc005c94a80 x1876699513061888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763547.933612:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699513061888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763547.933624:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763547.933629:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763547.933633:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513061888:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763547.933640:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763544/real 1789763547] req@ffff8dc005c97100 x1876699513062016/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763547.933653:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699513062016/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763547.933669:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763547.933674:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763547.933678:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513062016:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763547.933686:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763544/real 1789763547] req@ffff8dc005c97800 x1876699513062144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763547.933699:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699513062144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763547.933708:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763547.933713:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763547.933717:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513062144:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763547.933726:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763544/real 1789763547] req@ffff8dc005c95c00 x1876699513061760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763547.933739:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699513061760/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763547.933747:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763547.933750:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763547.933753:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513061760:192.168.203.156@tcp:250:kworker.0 00010000:00010000:0.0:1789763548.957070:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763549.981042:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763549.981053:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763549.981057:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763549.981068:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3524 00000100:00080000:3.0:1789763549.981079:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763549.981087:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763549.981122:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699513062272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1789763549.981148:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763549.981153:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763549.981159:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763549.981163:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763549.981171:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3524 00000100:00080000:3.0:1789763549.981177:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763549.981181:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763549.981192:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763549.981202:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513062272:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763549.981208:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699513062400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763549.981222:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763549.981225:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763549.981227:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763549.981231:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3524 00000100:00080000:3.0:1789763549.981235:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763549.981237:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763549.981243:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763549.981255:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699513062528/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763549.981264:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763549.981268:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763549.981269:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763549.981273:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3524 00000100:00080000:3.0:1789763549.981277:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763549.981284:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763549.981292:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763549.981299:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763549.981317:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699513062656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763549.981343:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763549/real 1789763549] req@ffff8dc0107aed80 x1876699513062272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763549.981372:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699513062272/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763549.981388:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763549.981394:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763549.981400:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513062272:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763549.981433:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513062400:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763549.981468:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763549.981488:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763549/real 1789763549] req@ffff8dc0107ac700 x1876699513062400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763549.981534:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699513062400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763549.981547:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763549.981559:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763549.981564:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513062400:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763549.981574:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513062528:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763549.981606:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763549.981619:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763549/real 1789763549] req@ffff8dc0107ad180 x1876699513062528/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763549.981631:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699513062528/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763549.981639:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763549.981642:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763549.981645:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513062528:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763549.981652:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513062656:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763549.981677:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763549.981693:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763549/real 1789763549] req@ffff8dc0107af480 x1876699513062656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763549.981709:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699513062656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763549.981720:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763549.981724:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763549.981729:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513062656:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763551.005173:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763552.029105:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763553.053131:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763554.077164:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763555.101100:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763555.101116:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763555.101124:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1789763555.101138:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763555.101144:0:6917:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3529 00000100:00080000:1.0:1789763555.101163:0:6917:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763555.101178:0:6917:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763555.101243:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699513062784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763555.101288:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763555.101293:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763555.101296:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763555.101301:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3529 00000100:00080000:1.0:1789763555.101306:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763555.101312:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763555.101327:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763555.101353:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699513062912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763555.101355:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513062784:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789763555.101385:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763555.101390:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763555.101392:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763555.101398:0:6917:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3529 00000100:00080000:1.0:1789763555.101403:0:6917:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763555.101408:0:6917:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763555.101414:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763555.101431:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699513063040/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763555.101445:0:6917:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763555.101450:0:6917:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763555.101453:0:6917:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763555.101481:0:6917:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3529 00000100:00080000:1.0:1789763555.101488:0:6917:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763555.101494:0:6917:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763555.101502:0:6917:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763555.101520:0:6917:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699513063168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1789763555.101534:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763555.101631:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763555/real 1789763555] req@ffff8dc005c95180 x1876699513062784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763555.101671:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699513062784/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763555.101688:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763555.101694:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763555.101700:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513062784:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763555.101737:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513062912:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763555.101772:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763555.101789:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763555/real 1789763555] req@ffff8dc005c95c00 x1876699513062912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763555.101805:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699513062912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763555.101816:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763555.101820:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763555.101824:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513062912:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763555.101831:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513063040:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789763555.101852:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763555.101864:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763555/real 1789763555] req@ffff8dc005c97800 x1876699513063040/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763555.101944:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699513063040/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763555.101957:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763555.101961:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763555.101965:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513063040:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763555.101972:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513063168:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763555.101999:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763555.102013:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763555/real 1789763555] req@ffff8dc005c97100 x1876699513063168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763555.102028:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699513063168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763555.102038:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763555.102042:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763555.102046:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513063168:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763556.125228:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763557.149038:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763558.173147:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763559.197062:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763560.221147:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:3.0:1789763560.221177:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763560.221179:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763560.221190:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763560.221209:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3534 00000100:00080000:0.0:1789763560.221242:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763560.221254:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763560.221337:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149500 x1876699513063296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763560.221392:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763560.221397:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763560.221400:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763560.221406:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3534 00000100:00080000:0.0:1789763560.221411:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763560.221416:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763560.221454:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763560.221503:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513063296:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763560.221505:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b100 x1876699513063424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763560.221535:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763560.221560:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763560.221565:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763560.221572:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3534 00000100:00080000:0.0:1789763560.221579:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763560.221586:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763560.221595:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763560.221615:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148e00 x1876699513063552/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763560.221631:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763560.221639:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1789763560.221641:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1789763560.221644:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763560.221655:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3534 00000100:00080000:0.0:1789763560.221661:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763560.221666:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763560.221691:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763560.221707:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514a300 x1876699513063680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763560.221736:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763560/real 1789763560] req@ffff8dc005149500 x1876699513063296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763560.221775:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149500 x1876699513063296/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763560.221790:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763560.221796:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763560.221801:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513063296:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763560.221852:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513063424:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763560.221878:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763560.221892:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763560/real 1789763560] req@ffff8dc00514b100 x1876699513063424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763560.221906:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b100 x1876699513063424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763560.221915:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763560.221919:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763560.221923:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513063424:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763560.221932:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513063552:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789763560.222002:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763560.222026:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763560/real 1789763560] req@ffff8dc005148e00 x1876699513063552/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763560.222041:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148e00 x1876699513063552/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763560.222060:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763560.222066:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763560.222072:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513063552:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763560.222082:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513063680:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763560.222119:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763560.222140:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763560/real 1789763560] req@ffff8dc00514a300 x1876699513063680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763560.222158:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514a300 x1876699513063680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763560.222167:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763560.222171:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763560.222174:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513063680:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763561.245177:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763562.269119:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763563.293137:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763564.317129:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763565.341066:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:1.0:1789763565.341077:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763565.341079:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763565.341087:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763565.341103:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3539 00000100:00080000:2.0:1789763565.341120:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763565.341132:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763565.341220:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699513063808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763565.341279:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763565.341286:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763565.341289:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763565.341298:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3539 00000100:00080000:2.0:1789763565.341309:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763565.341336:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763565.341355:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763565.341380:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699513063936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763565.341425:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513063808:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763565.341435:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763565.341442:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763565.341444:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763565.341450:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3539 00000100:00080000:2.0:1789763565.341456:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763565.341461:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763565.341469:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763565.341491:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699513064064/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763565.341506:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763565.341513:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763565.341516:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763565.341521:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3539 00000800:00000100:0.0:1789763565.341553:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789763565.341613:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763565.341626:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763565.341637:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763565.341662:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699513064192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763565.341684:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763565/real 1789763565] req@ffff8dc00368c700 x1876699513063808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763565.341724:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699513063808/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763565.341746:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763565.341754:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763565.341764:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513063808:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763565.341812:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513063936:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763565.341863:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763565.341897:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763565/real 1789763565] req@ffff8dc00368d180 x1876699513063936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763565.341973:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699513063936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763565.341993:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763565.342002:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763565.342010:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513063936:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763565.342031:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513064064:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763565.342099:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763565.342144:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763565/real 1789763565] req@ffff8dc00368e680 x1876699513064064/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763565.342167:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699513064064/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763565.342182:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763565.342187:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763565.342194:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513064064:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763565.342204:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513064192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763565.342243:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763565.342272:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763565/real 1789763565] req@ffff8dc00368dc00 x1876699513064192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763565.342295:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699513064192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763565.342329:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763565.342336:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763565.342342:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513064192:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763566.365095:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763567.389120:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763568.413193:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763569.437145:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763570.461078:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763570.461091:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763570.461097:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763570.461111:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3545 00010000:00010000:1.0:1789763570.461118:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763570.461124:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763570.461134:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763570.461195:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699513064320/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763570.461226:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763570.461229:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763570.461231:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763570.461235:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3545 00000100:00080000:3.0:1789763570.461239:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763570.461243:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763570.461255:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763570.461277:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513064320:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763570.461278:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699513064448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763570.461301:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763570.461306:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763570.461309:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763570.461315:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3545 00000100:00080000:3.0:1789763570.461320:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763570.461326:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763570.461334:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763570.461373:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699513064576/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763570.461389:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763570.461389:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763570.461395:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763570.461397:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763570.461414:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3545 00000100:00080000:3.0:1789763570.461420:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763570.461424:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763570.461430:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763570.461443:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699513064704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763570.461464:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763570/real 1789763570] req@ffff8dc0107ad500 x1876699513064320/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763570.461490:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699513064320/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763570.461503:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763570.461518:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763570.461522:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513064320:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763570.461549:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513064448:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763570.461571:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763570.461581:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763570/real 1789763570] req@ffff8dc0107ac000 x1876699513064448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763570.461591:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699513064448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763570.461599:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763570.461602:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763570.461604:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513064448:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763570.461609:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513064576:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763570.461624:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763570.461631:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763570/real 1789763570] req@ffff8dc0107afb80 x1876699513064576/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763570.461656:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699513064576/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763570.461667:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763570.461672:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763570.461676:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513064576:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763570.461685:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513064704:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763570.461713:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763570.461728:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763570/real 1789763570] req@ffff8dc0107ae300 x1876699513064704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763570.461744:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699513064704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763570.461756:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763570.461760:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763570.461764:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513064704:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763571.485203:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763572.509167:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763573.533134:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763574.557177:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763575.581047:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763575.581060:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763575.581066:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763575.581083:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3550 00000100:00080000:3.0:1789763575.581117:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763575.581128:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763575.581156:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763575.581197:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699513064832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763575.581244:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763575.581250:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763575.581253:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763575.581258:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3550 00000100:00080000:3.0:1789763575.581265:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763575.581271:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763575.581286:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763575.581337:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513064832:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763575.581339:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699513064960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763575.581361:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763575.581364:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763575.581366:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763575.581370:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3550 00000100:00080000:3.0:1789763575.581374:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763575.581377:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763575.581400:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763575.581425:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699513065088/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763575.581445:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763575.581452:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763575.581456:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763575.581464:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3550 00000100:00080000:3.0:1789763575.581473:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763575.581483:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789763575.581494:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789763575.581497:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763575.581525:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699513065216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763575.581574:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763575/real 1789763575] req@ffff8dc0107aca80 x1876699513064832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763575.581619:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699513064832/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763575.581636:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763575.581642:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763575.581648:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513064832:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763575.581708:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513064960:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763575.581749:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763575.581768:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763575/real 1789763575] req@ffff8dc0107adc00 x1876699513064960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763575.581784:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699513064960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763575.581795:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763575.581800:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763575.581804:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513064960:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763575.581811:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513065088:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763575.581849:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763575.581858:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763575/real 1789763575] req@ffff8dc0107ae300 x1876699513065088/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763575.581868:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699513065088/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763575.581875:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763575.581879:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763575.581899:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513065088:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763575.581906:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513065216:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763575.581927:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763575.581937:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763575/real 1789763575] req@ffff8dc0107afb80 x1876699513065216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763575.581950:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699513065216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763575.581968:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763575.581972:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763575.581975:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513065216:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763576.605196:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763577.629257:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763578.653161:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763579.677165:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763580.701106:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763580.701124:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763580.701132:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763580.701149:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3555 00000100:00080000:2.0:1789763580.701166:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:1.0:1789763580.701169:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763580.701181:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763580.701242:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699513065344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763580.701300:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763580.701308:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763580.701312:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763580.701321:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3555 00000100:00080000:2.0:1789763580.701329:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763580.701338:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763580.701353:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763580.701361:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513065344:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763580.701378:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699513065472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763580.701397:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763580.701403:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763580.701406:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763580.701413:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3555 00000100:00080000:2.0:1789763580.701420:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763580.701426:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763580.701434:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763580.701452:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699513065600/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763580.701456:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789763580.701471:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763580.701476:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763580.701479:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763580.701486:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3555 00000100:00080000:2.0:1789763580.701492:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763580.701499:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763580.701508:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763580.701525:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699513065728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763580.701532:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763580/real 1789763580] req@ffff8dc00368df80 x1876699513065344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763580.701568:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699513065344/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763580.701590:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763580.701598:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763580.701606:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513065344:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763580.701646:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513065472:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763580.701682:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763580.701700:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763580/real 1789763580] req@ffff8dc00368f800 x1876699513065472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763580.701716:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699513065472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763580.701728:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763580.701732:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763580.701736:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513065472:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763580.701763:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513065600:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763580.701788:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763580.701802:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763580/real 1789763580] req@ffff8dc00368e300 x1876699513065600/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763580.701817:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699513065600/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763580.701827:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763580.701831:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763580.701835:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513065600:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763580.701841:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513065728:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763580.701861:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763580.701872:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763580/real 1789763580] req@ffff8dc00368ed80 x1876699513065728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763580.701886:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699513065728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763580.701896:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763580.701932:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763580.701937:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513065728:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763581.725163:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763582.749160:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763583.773096:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763584.797115:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763585.821075:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:1.0:1789763585.821082:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763585.821091:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763585.821098:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763585.821113:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3560 00000100:00080000:3.0:1789763585.821128:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763585.821137:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763585.821192:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699513065856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763585.821229:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763585.821252:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763585.821255:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763585.821261:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3560 00000100:00080000:3.0:1789763585.821265:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763585.821291:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763585.821306:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763585.821326:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513065856:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763585.821331:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699513065984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763585.821353:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763585.821357:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763585.821360:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763585.821366:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3560 00000100:00080000:3.0:1789763585.821371:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763585.821375:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763585.821395:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763585.821411:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699513066112/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763585.821425:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763585.821430:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763585.821433:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763585.821437:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3560 00000100:00080000:3.0:1789763585.821442:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763585.821446:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763585.821451:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763585.821454:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763585.821470:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699513066240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763585.821555:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763585/real 1789763585] req@ffff8dc0107ac000 x1876699513065856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763585.821599:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699513065856/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763585.821623:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763585.821632:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763585.821641:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513065856:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763585.821694:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513065984:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763585.821726:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763585.821742:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763585/real 1789763585] req@ffff8dc0107ad500 x1876699513065984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763585.821757:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699513065984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763585.821767:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763585.821771:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763585.821774:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513065984:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763585.821781:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513066112:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763585.821801:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763585.821811:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763585/real 1789763585] req@ffff8dc0107af480 x1876699513066112/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763585.821824:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699513066112/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763585.821846:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763585.821852:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763585.821858:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513066112:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763585.821869:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513066240:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763585.821903:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763585.821957:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763585/real 1789763585] req@ffff8dc0107ad180 x1876699513066240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763585.821972:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699513066240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763585.821981:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763585.821985:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763585.821989:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513066240:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763586.845181:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763587.869252:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763588.893106:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763589.917137:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763590.941063:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763590.941073:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763590.941078:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763590.941110:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3565 00000100:00080000:3.0:1789763590.941129:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763590.941140:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763590.941167:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763590.941192:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699513066368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763590.941277:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763590.941281:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763590.941283:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763590.941287:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3565 00000100:00080000:3.0:1789763590.941292:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763590.941297:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763590.941308:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763590.941324:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513066368:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763590.941326:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699513066496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763590.941344:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763590.941347:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763590.941349:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763590.941354:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3565 00000100:00080000:3.0:1789763590.941357:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763590.941360:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763590.941365:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763590.941377:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699513066624/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763590.941387:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763590.941391:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763590.941393:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763590.941396:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3565 00000100:00080000:3.0:1789763590.941399:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789763590.941402:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763590.941408:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763590.941415:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763590.941427:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699513066752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763590.941473:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763590/real 1789763590] req@ffff8dc0107ac700 x1876699513066368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763590.941496:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699513066368/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763590.941527:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763590.941533:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763590.941538:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513066368:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763590.941569:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513066496:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763590.941613:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763590.941627:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763590/real 1789763590] req@ffff8dc0107aed80 x1876699513066496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763590.941641:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699513066496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763590.941650:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763590.941672:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763590.941676:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513066496:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763590.941683:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513066624:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763590.941705:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763590.941734:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763590/real 1789763590] req@ffff8dc0107adf80 x1876699513066624/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763590.941748:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699513066624/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763590.941758:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763590.941761:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763590.941765:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513066624:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763590.941771:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513066752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763590.941790:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763590.941801:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763590/real 1789763590] req@ffff8dc0107ac380 x1876699513066752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763590.941814:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699513066752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763590.941824:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763590.941827:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763590.941830:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513066752:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763591.965147:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763592.989167:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763594.013433:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763595.037228:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763596.061046:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763596.061059:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763596.061065:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763596.061079:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3570 00000100:00080000:2.0:1789763596.061093:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763596.061103:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763596.061127:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763596.061186:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699513066880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763596.061224:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763596.061230:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763596.061233:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763596.061239:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3570 00000100:00080000:2.0:1789763596.061245:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763596.061250:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763596.061264:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763596.061287:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699513067008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763596.061296:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513066880:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763596.061317:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763596.061323:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763596.061326:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763596.061332:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3570 00000100:00080000:2.0:1789763596.061337:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763596.061341:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763596.061347:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763596.061366:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699513067136/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763596.061380:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763596.061385:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763596.061388:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763596.061392:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3570 00000800:00000100:0.0:1789763596.061401:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789763596.061402:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763596.061413:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763596.061421:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763596.061437:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699513067264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763596.061483:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763596/real 1789763596] req@ffff8dc00368ca80 x1876699513066880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763596.061510:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699513066880/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763596.061524:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763596.061529:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763596.061534:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513066880:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763596.061562:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513067008:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763596.061590:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763596.061621:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763596/real 1789763596] req@ffff8dc00368ea00 x1876699513067008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763596.061647:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699513067008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763596.061675:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763596.061681:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763596.061688:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513067008:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763596.061701:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513067136:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763596.061737:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763596.061752:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763596/real 1789763596] req@ffff8dc00368c000 x1876699513067136/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763596.061766:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699513067136/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763596.061776:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763596.061780:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763596.061784:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513067136:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763596.061791:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513067264:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763596.061810:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763596.061821:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763596/real 1789763596] req@ffff8dc00368ce00 x1876699513067264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763596.061834:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699513067264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763596.061843:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763596.061846:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763596.061850:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513067264:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763597.085110:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763598.109135:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763599.133209:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763600.157213:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763601.181106:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763601.181137:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763601.181143:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763601.181156:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3575 00000100:00080000:2.0:1789763601.181174:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763601.181186:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789763601.181204:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763601.181240:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699513067392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763601.181274:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763601.181281:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763601.181284:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763601.181292:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3575 00000100:00080000:2.0:1789763601.181297:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763601.181302:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763601.181315:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763601.181329:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699513067520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763601.181347:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513067392:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763601.181351:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763601.181364:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763601.181374:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763601.181380:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3575 00000100:00080000:2.0:1789763601.181384:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763601.181387:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763601.181393:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763601.181404:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699513067648/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763601.181415:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763601.181420:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763601.181422:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763601.181427:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3575 00000100:00080000:2.0:1789763601.181432:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763601.181437:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763601.181441:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763601.181454:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699513067776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763601.181501:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763601.181584:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763601/real 1789763601] req@ffff8dc00368f100 x1876699513067392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763601.181615:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699513067392/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763601.181632:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763601.181640:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763601.181646:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513067392:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763601.181693:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513067520:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763601.181740:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763601.181762:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763601/real 1789763601] req@ffff8dc00368f480 x1876699513067520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763601.181779:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699513067520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763601.181791:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763601.181796:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763601.181800:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513067520:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763601.181809:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513067648:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763601.181844:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763601.181861:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763601/real 1789763601] req@ffff8dc00368fb80 x1876699513067648/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763601.181890:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699513067648/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763601.181965:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763601.181971:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763601.181975:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513067648:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763601.181987:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513067776:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763601.182023:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763601.182044:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763601/real 1789763601] req@ffff8dc00368d880 x1876699513067776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763601.182072:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699513067776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763601.182083:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763601.182089:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763601.182099:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513067776:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763602.205145:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763603.229177:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763604.253073:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763605.277210:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763606.301106:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763606.301124:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763606.301132:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763606.301147:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3580 00010000:00010000:1.0:1789763606.301159:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763606.301161:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763606.301174:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763606.301224:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699513067904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763606.301261:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763606.301267:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763606.301269:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763606.301275:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3580 00000100:00080000:3.0:1789763606.301280:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763606.301286:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763606.301299:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763606.301317:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699513068032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763606.301331:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513067904:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763606.301339:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763606.301346:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763606.301348:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763606.301354:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3580 00000100:00080000:3.0:1789763606.301358:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763606.301363:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763606.301384:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763606.301403:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699513068160/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763606.301418:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763606.301422:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763606.301425:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763606.301443:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3580 00000100:00080000:3.0:1789763606.301450:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789763606.301459:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763606.301461:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763606.301473:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763606.301488:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699513068288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763606.301552:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763606/real 1789763606] req@ffff8dc0107af800 x1876699513067904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763606.301588:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699513067904/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763606.301603:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763606.301608:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763606.301614:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513067904:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763606.301644:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513068032:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763606.301687:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763606.301714:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763606/real 1789763606] req@ffff8dc0107ae680 x1876699513068032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763606.301742:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699513068032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763606.301760:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763606.301766:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763606.301771:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513068032:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763606.301780:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513068160:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763606.301812:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763606.301828:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763606/real 1789763606] req@ffff8dc0107ace00 x1876699513068160/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763606.301844:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699513068160/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763606.301862:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763606.301866:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763606.301870:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513068160:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763606.301877:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513068288:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763606.301929:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763606.301946:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763606/real 1789763606] req@ffff8dc0107af100 x1876699513068288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763606.301961:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699513068288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763606.301972:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763606.301976:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763606.301980:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513068288:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763607.325134:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763608.349141:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763609.373161:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763610.397134:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763611.421135:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763611.421163:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763611.421170:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1789763611.421173:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763611.421189:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3586 00000100:00080000:2.0:1789763611.421207:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763611.421217:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763611.421286:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699513068416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763611.421332:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763611.421337:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763611.421340:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763611.421345:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3586 00000100:00080000:2.0:1789763611.421361:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763611.421368:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763611.421383:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763611.421413:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513068416:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763611.421413:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699513068544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763611.421443:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763611.421448:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763611.421450:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763611.421458:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3586 00000100:00080000:2.0:1789763611.421463:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763611.421467:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763611.421474:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763611.421489:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699513068672/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763611.421503:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763611.421508:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763611.421511:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763611.421516:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3586 00000100:00080000:2.0:1789763611.421522:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763611.421527:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763611.421534:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763611.421550:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699513068800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763611.421622:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513068544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763611.421667:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513068672:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763611.421707:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513068800:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763612.445151:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763613.469130:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763614.493267:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789763614.493419:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1789763614.493489:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789763614.493514:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789763614.493571:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789763614.493584:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789763614.493596:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1789763614.493660:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763611/real 1789763614] req@ffff8dc00368fb80 x1876699513068544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763614.493736:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699513068544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763614.493759:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763614.493769:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763614.493777:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513068544:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763614.493792:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763611/real 1789763614] req@ffff8dc00368f480 x1876699513068672/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763614.493817:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699513068672/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763614.493831:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763614.493839:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763614.493845:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513068672:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763614.493860:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763611/real 1789763614] req@ffff8dc00368f100 x1876699513068800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763614.493950:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699513068800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763614.493970:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763614.493977:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763614.493985:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513068800:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763614.493999:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763611/real 1789763614] req@ffff8dc00368d880 x1876699513068416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763614.494059:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699513068416/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763614.494078:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763614.494088:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763614.494095:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513068416:192.168.203.156@tcp:250:kworker.0 00010000:00010000:1.0:1789763615.517221:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763616.541083:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763616.541100:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763616.541106:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763616.541127:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3591 00010000:00010000:0.0:1789763616.541132:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763616.541146:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763616.541162:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763616.541226:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699513068928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763616.541272:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763616.541281:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763616.541283:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763616.541288:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3591 00000100:00080000:2.0:1789763616.541293:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763616.541297:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763616.541311:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763616.541325:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699513069056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763616.541338:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763616.541342:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763616.541344:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763616.541349:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3591 00000100:00080000:2.0:1789763616.541352:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763616.541358:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763616.541363:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763616.541375:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699513069184/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763616.541388:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763616.541398:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513068928:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763616.541399:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763616.541406:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763616.541422:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3591 00000100:00080000:2.0:1789763616.541443:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763616.541447:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763616.541452:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763616.541465:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699513069312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763616.541525:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763616.541586:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763616/real 1789763616] req@ffff8dc00368d880 x1876699513068928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763616.541607:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699513068928/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763616.541620:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763616.541628:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763616.541636:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513068928:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763616.541648:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513069056:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763616.541679:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763616.541698:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763616/real 1789763616] req@ffff8dc00368f100 x1876699513069056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763616.541717:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699513069056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763616.541730:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763616.541734:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763616.541740:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513069056:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763616.541799:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513069184:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763616.541833:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763616.541851:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763616/real 1789763616] req@ffff8dc00368f480 x1876699513069184/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763616.541867:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699513069184/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763616.541877:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763616.541881:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763616.541886:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513069184:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763616.541894:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513069312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763616.541961:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763616.541977:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763616/real 1789763616] req@ffff8dc00368fb80 x1876699513069312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763616.541996:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699513069312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763616.542024:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763616.542028:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763616.542033:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513069312:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763617.565134:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763618.589199:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763619.613092:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763620.637170:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763621.661099:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763621.661121:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763621.661130:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763621.661152:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3596 00010000:00010000:1.0:1789763621.661163:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763621.661173:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763621.661196:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763621.661273:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699513069440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763621.661340:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763621.661348:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763621.661351:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763621.661359:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3596 00000100:00080000:3.0:1789763621.661369:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763621.661377:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763621.661395:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763621.661408:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513069440:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763621.661423:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699513069568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763621.661443:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763621.661449:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763621.661463:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763621.661469:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3596 00000100:00080000:3.0:1789763621.661474:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763621.661479:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763621.661489:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763621.661510:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699513069696/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763621.661529:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763621.661535:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763621.661544:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763621.661547:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763621.661554:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3596 00000100:00080000:3.0:1789763621.661572:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763621.661578:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763621.661586:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763621.661604:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699513069824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763621.661621:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763621/real 1789763621] req@ffff8dc0107aea00 x1876699513069440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763621.661692:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699513069440/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763621.661710:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763621.661716:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763621.661722:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513069440:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763621.661757:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513069568:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763621.661794:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763621.661812:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763621/real 1789763621] req@ffff8dc0107ad880 x1876699513069568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763621.661828:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699513069568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763621.661839:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763621.661844:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763621.661848:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513069568:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763621.661856:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513069696:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763621.661878:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763621.661890:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763621/real 1789763621] req@ffff8dc0107af100 x1876699513069696/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763621.661951:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699513069696/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763621.661962:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763621.661991:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763621.661997:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513069696:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763621.662007:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513069824:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763621.662038:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763621.662053:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763621/real 1789763621] req@ffff8dc0107ace00 x1876699513069824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763621.662068:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699513069824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763621.662079:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763621.662083:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763621.662087:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513069824:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763622.685213:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763623.709181:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763624.733118:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763625.757160:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763626.781052:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763626.781065:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763626.781072:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763626.781090:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3601 00000100:00080000:2.0:1789763626.781107:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763626.781117:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763626.781172:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699513069952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1789763626.781212:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763626.781213:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763626.781225:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763626.781230:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763626.781237:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3601 00000100:00080000:2.0:1789763626.781245:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763626.781251:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763626.781265:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763626.781283:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699513070080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763626.781299:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513069952:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763626.781304:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763626.781324:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763626.781327:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763626.781334:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3601 00000100:00080000:2.0:1789763626.781339:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763626.781346:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763626.781352:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763626.781369:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699513070208/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763626.781382:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763626.781387:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763626.781391:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763626.781396:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3601 00000100:00080000:2.0:1789763626.781401:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763626.781405:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763626.781412:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763626.781424:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699513070336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763626.781442:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763626.781524:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763626/real 1789763626] req@ffff8dc00368ce00 x1876699513069952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763626.781555:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699513069952/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763626.781584:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763626.781591:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763626.781597:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513069952:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763626.781636:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513070080:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763626.781665:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763626.781681:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763626/real 1789763626] req@ffff8dc00368c000 x1876699513070080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763626.781697:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699513070080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763626.781708:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763626.781713:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763626.781717:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513070080:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763626.781724:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513070208:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763626.781747:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763626.781759:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763626/real 1789763626] req@ffff8dc00368ea00 x1876699513070208/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763626.781773:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699513070208/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763626.781784:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763626.781787:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763626.781791:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513070208:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763626.781799:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513070336:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763626.781820:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763626.781831:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763626/real 1789763626] req@ffff8dc00368ca80 x1876699513070336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763651 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763626.781850:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699513070336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763651 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763626.781866:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763626.781873:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763626.781880:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513070336:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763627.805152:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763628.829161:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763629.853182:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763630.877210:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763631.901157:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763631.901174:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763631.901182:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763631.901199:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3606 00000100:00080000:3.0:1789763631.901217:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763631.901229:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763631.901240:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763631.901298:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699513070464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763631.901352:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763631.901360:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763631.901364:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763631.901373:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3606 00000100:00080000:3.0:1789763631.901395:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763631.901404:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763631.901425:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763631.901441:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513070464:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763631.901458:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699513070592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763631.901479:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763631.901484:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763631.901487:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763631.901493:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3606 00000100:00080000:3.0:1789763631.901500:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763631.901505:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763631.901515:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763631.901533:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699513070720/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763631.901548:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763631.901553:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763631.901555:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763631.901561:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3606 00000100:00080000:3.0:1789763631.901569:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789763631.901572:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763631.901579:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763631.901588:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763631.901601:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699513070848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763631.901671:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763631/real 1789763631] req@ffff8dc0107ae680 x1876699513070464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763631.901703:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699513070464/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763631.901720:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763631.901725:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763631.901731:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513070464:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763631.901768:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513070592:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763631.901804:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763631.901839:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763631/real 1789763631] req@ffff8dc0107af800 x1876699513070592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763631.901865:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699513070592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763631.901883:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763631.901889:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763631.901962:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513070592:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763631.901977:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513070720:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763631.902016:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763631.902034:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763631/real 1789763631] req@ffff8dc0107ac380 x1876699513070720/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763631.902050:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699513070720/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763631.902062:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763631.902066:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763631.902071:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513070720:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763631.902079:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513070848:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763631.902102:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763631.902114:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763631/real 1789763631] req@ffff8dc0107adf80 x1876699513070848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763631.902128:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699513070848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763631.902148:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763631.902152:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763631.902156:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513070848:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763632.925109:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763633.949186:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763634.973116:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763635.997117:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763637.021053:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763637.021084:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763637.021091:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763637.021111:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3611 00000100:00080000:3.0:1789763637.021132:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763637.021144:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789763637.021162:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763637.021212:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699513070976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763637.021268:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763637.021277:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763637.021281:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763637.021290:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3611 00000100:00080000:3.0:1789763637.021301:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763637.021313:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763637.021333:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763637.021339:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513070976:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763637.021363:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699513071104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763637.021384:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763637.021389:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763637.021392:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763637.021399:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3611 00000100:00080000:3.0:1789763637.021406:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763637.021412:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763637.021421:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763637.021440:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699513071232/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763637.021444:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763637.021470:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763637.021475:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763637.021478:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763637.021486:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3611 00000100:00080000:3.0:1789763637.021490:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763637.021500:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763637.021510:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763637.021526:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763637/real 1789763637] req@ffff8dc0107aed80 x1876699513070976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763637.021531:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699513071360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763637.021634:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699513070976/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763637.021707:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763637.021716:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763637.021724:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513070976:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763637.021767:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513071104:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763637.021814:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763637.021836:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763637/real 1789763637] req@ffff8dc0107ac700 x1876699513071104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763637.021852:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699513071104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763637.021886:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763637.021890:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763637.021896:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513071104:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763637.021936:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513071232:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763637.021973:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763637.021997:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763637/real 1789763637] req@ffff8dc0107ad180 x1876699513071232/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763637.022014:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699513071232/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763637.022032:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763637.022039:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763637.022047:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513071232:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763637.022064:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513071360:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763637.022096:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763637.022115:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763637/real 1789763637] req@ffff8dc0107af480 x1876699513071360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763637.022134:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699513071360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763637.022149:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763637.022154:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763637.022161:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513071360:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763638.045147:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763639.069174:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763640.093096:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763641.117131:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763642.141100:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763642.141114:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763642.141121:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763642.141137:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3616 00000100:00080000:3.0:1789763642.141153:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763642.141163:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763642.141206:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763642.141215:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699513071488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763642.141260:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763642.141283:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763642.141290:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763642.141301:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3616 00000100:00080000:3.0:1789763642.141310:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763642.141321:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763642.141342:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763642.141343:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513071488:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763642.141395:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699513071616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763642.141427:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763642.141435:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763642.141440:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763642.141449:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3616 00000100:00080000:3.0:1789763642.141457:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763642.141467:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763642.141484:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789763642.141494:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763642.141533:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699513071744/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763642.141571:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763642/real 1789763642] req@ffff8dc0107ad500 x1876699513071488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763642.141586:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763642.141611:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763642.141622:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763642.141647:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3616 00000100:00080000:3.0:1789763642.141659:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763642.141666:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699513071488/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763642.141678:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763642.141706:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789763642.141712:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763642.141721:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763642.141731:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513071488:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763642.141738:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699513071872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763642.141807:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513071616:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763642.141879:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763642.141933:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763642/real 1789763642] req@ffff8dc0107ac000 x1876699513071616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763642.141953:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699513071616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763642.141966:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763642.141970:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763642.141975:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513071616:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763642.141984:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513071744:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763642.142014:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763642.142028:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763642/real 1789763642] req@ffff8dc0107afb80 x1876699513071744/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763642.142044:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699513071744/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763642.142063:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763642.142067:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763642.142071:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513071744:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763642.142078:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513071872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763642.142099:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763642.142111:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763642/real 1789763642] req@ffff8dc0107ae300 x1876699513071872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763642.142125:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699513071872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763642.142135:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763642.142139:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763642.142142:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513071872:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763643.165142:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763644.189216:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763645.213154:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763646.237116:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763647.261052:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763647.261071:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763647.261079:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763647.261099:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3621 00000100:00080000:3.0:1789763647.261117:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763647.261132:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763647.261202:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699513072000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763647.261261:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763647.261279:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763647.261283:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763647.261292:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3621 00000100:00080000:3.0:1789763647.261301:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763647.261310:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763647.261328:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763647.261355:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513072000:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763647.261360:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699513072128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1789763647.261378:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.0:1789763647.261489:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763647.261562:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763647/real 1789763647] req@ffff8dc0107adc00 x1876699513072000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763647.261606:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699513072000/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763647.261621:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763647.261634:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763647.261637:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763647.261642:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763647.261646:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3621 00000100:00080000:2.0:1789763647.261654:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:3.0:1789763647.261655:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763647.261668:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513072000:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763647.261670:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763647.261686:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763647.261707:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699513072256/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763647.261726:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513072128:192.168.203.156@tcp:38:kworker.0 00000100:00080000:3.0:1789763647.261741:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763647.261746:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763647.261749:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763647.261755:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3621 00000100:00080000:3.0:1789763647.261772:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789763647.261803:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763647.261808:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763647.261827:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763647.261835:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763647/real 1789763647] req@ffff8dc0107aca80 x1876699513072128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763647.261850:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699513072384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763647.261859:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699513072128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763647.261947:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763647.261952:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763647.261958:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513072128:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763647.261981:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513072256:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789763647.262009:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763647.262023:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763647/real 1789763647] req@ffff8dc0107ae300 x1876699513072256/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763647.262037:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699513072256/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763647.262054:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763647.262058:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763647.262062:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513072256:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763647.262068:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513072384:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763647.262088:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763647.262103:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763647/real 1789763647] req@ffff8dc0107afb80 x1876699513072384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763647.262123:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699513072384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763647.262154:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763647.262161:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763647.262167:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513072384:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763648.285159:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763649.308999:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763650.333105:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763651.357206:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763652.381041:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763652.381056:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763652.381062:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1789763652.381067:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763652.381075:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3626 00000100:00080000:2.0:1789763652.381087:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763652.381095:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763652.381140:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699513072512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763652.381167:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763652.381173:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763652.381190:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763652.381198:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3626 00000100:00080000:2.0:1789763652.381205:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763652.381214:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763652.381242:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763652.381257:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513072512:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763652.381268:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699513072640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763652.381287:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763652.381293:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763652.381296:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763652.381303:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3626 00000100:00080000:2.0:1789763652.381310:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763652.381316:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763652.381325:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763652.381343:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699513072768/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763652.381360:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763652.381366:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763652.381370:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789763652.381374:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789763652.381376:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3626 00000100:00080000:2.0:1789763652.381397:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763652.381410:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763652.381417:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763652.381428:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699513072896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763652.381455:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763652/real 1789763652] req@ffff8dc00368ed80 x1876699513072512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763652.381480:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699513072512/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763652.381492:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763652.381498:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763652.381502:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513072512:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763652.381528:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513072640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763652.381549:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763652.381561:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763652/real 1789763652] req@ffff8dc00368e300 x1876699513072640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763652.381571:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699513072640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763652.381577:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763652.381580:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763652.381583:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513072640:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763652.381586:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513072768:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763652.381599:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763652.381606:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763652/real 1789763652] req@ffff8dc00368f800 x1876699513072768/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763652.381615:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699513072768/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763652.381621:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763652.381624:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763652.381626:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513072768:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763652.381629:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513072896:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763652.381641:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763652.381647:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763652/real 1789763652] req@ffff8dc00368df80 x1876699513072896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763652.381656:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699513072896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763652.381662:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763652.381664:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763652.381666:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513072896:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763653.405140:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763654.429140:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763655.453054:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763656.477151:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763657.501045:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763657.501067:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763657.501077:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763657.501098:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3632 00010000:00010000:1.0:1789763657.501117:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763657.501118:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763657.501140:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763657.501232:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699513073024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763657.501287:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763657.501295:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763657.501300:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763657.501311:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3632 00000100:00080000:2.0:1789763657.501320:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763657.501332:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763657.501341:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513073024:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789763657.501354:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763657.501392:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699513073152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763657.501418:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1789763657.501428:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789763657.501435:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763657.501441:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763657.501453:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3632 00000100:00080000:2.0:1789763657.501465:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763657.501476:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763657.501488:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763657.501493:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763657/real 1789763657] req@ffff8dc00368dc00 x1876699513073024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763657.501536:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699513073280/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763657.501545:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699513073024/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763657.501631:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763657.501641:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00080000:2.0:1789763657.501642:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763657.501653:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513073024:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763657.501659:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763657.501668:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763657.501680:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3632 00000100:00080000:2.0:1789763657.501690:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763657.501700:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763657.501732:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513073152:192.168.203.156@tcp:38:kworker.0 00000008:00100000:2.0:1789763657.501757:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763657.501805:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763657.501806:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699513073408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763657.501829:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763657/real 1789763657] req@ffff8dc00368e680 x1876699513073152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763657.501857:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699513073152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763657.501882:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763657.501890:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763657.501941:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513073152:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763657.501958:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513073280:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763657.502013:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763657.502029:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763657/real 1789763657] req@ffff8dc00368d180 x1876699513073280/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763657.502045:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699513073280/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763657.502056:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763657.502061:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763657.502065:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513073280:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763657.502082:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513073408:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763657.502111:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763657.502122:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763657/real 1789763657] req@ffff8dc00368c700 x1876699513073408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763657.502132:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699513073408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763657.502139:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763657.502141:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763657.502144:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513073408:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763658.525153:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763659.549149:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763660.573174:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763661.597127:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763662.621053:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763662.621075:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763662.621091:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763662.621098:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763662.621111:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3637 00000100:00080000:2.0:1789763662.621126:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763662.621135:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763662.621194:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699513073536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763662.621232:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763662.621240:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763662.621243:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763662.621250:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3637 00000100:00080000:2.0:1789763662.621256:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763662.621262:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763662.621276:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763662.621278:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513073536:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763662.621296:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699513073664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763662.621324:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763662.621329:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763662.621331:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763662.621336:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3637 00000100:00080000:2.0:1789763662.621341:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763662.621346:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763662.621352:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763662.621384:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699513073792/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763662.621403:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789763662.621432:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763662.621438:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763662.621442:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763662.621451:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3637 00000100:00080000:2.0:1789763662.621461:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763662.621468:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763662.621477:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763662.621502:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699513073920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763662.621523:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763662/real 1789763662] req@ffff8dc00368c700 x1876699513073536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763662.621572:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699513073536/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763662.621587:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763662.621593:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763662.621599:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513073536:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763662.621648:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513073664:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763662.621718:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763662.621759:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763662/real 1789763662] req@ffff8dc00368d180 x1876699513073664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763662.621768:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699513073664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763662.621774:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763662.621777:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763662.621779:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513073664:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763662.621784:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513073792:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763662.621798:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763662.621825:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763662/real 1789763662] req@ffff8dc00368e680 x1876699513073792/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763662.621844:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699513073792/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763662.621850:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763662.621853:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763662.621855:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513073792:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763662.621859:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513073920:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763662.621894:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763662.621907:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763662/real 1789763662] req@ffff8dc00368dc00 x1876699513073920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763662.621921:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699513073920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763662.621930:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763662.621934:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763662.621937:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513073920:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763663.645151:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763664.669143:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763665.693125:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763666.717224:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763667.741083:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763667.741099:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763667.741106:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763667.741124:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3642 00010000:00010000:2.0:1789763667.741125:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763667.741143:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763667.741156:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763667.741212:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699513074048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763667.741249:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763667.741254:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763667.741257:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763667.741263:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3642 00000100:00080000:3.0:1789763667.741267:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763667.741271:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763667.741292:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763667.741311:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699513074176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763667.741327:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513074048:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763667.741332:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763667.741345:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763667.741349:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763667.741355:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3642 00000100:00080000:3.0:1789763667.741359:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763667.741364:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763667.741371:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763667.741389:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699513074304/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763667.741404:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763667.741409:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763667.741411:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763667.741415:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3642 00000100:00080000:3.0:1789763667.741420:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763667.741425:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763667.741431:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763667.741445:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699513074432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763667.741461:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763667.741550:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763667/real 1789763667] req@ffff8dc0107ac000 x1876699513074048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763667.741583:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699513074048/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763667.741600:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763667.741605:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763667.741611:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513074048:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763667.741671:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513074176:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763667.741708:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763667.741725:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763667/real 1789763667] req@ffff8dc0107ad500 x1876699513074176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763667.741742:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699513074176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763667.741753:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763667.741758:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763667.741762:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513074176:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763667.741770:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513074304:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763667.741792:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763667.741804:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763667/real 1789763667] req@ffff8dc0107af480 x1876699513074304/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763667.741818:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699513074304/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763667.741828:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763667.741832:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763667.741836:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513074304:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763667.741843:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513074432:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763667.741864:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763667.741876:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763667/real 1789763667] req@ffff8dc0107ad180 x1876699513074432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763667.741890:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699513074432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763667.741935:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763667.741939:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763667.741943:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513074432:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763668.765163:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763669.789111:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763670.813136:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763671.837148:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763672.861137:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763672.861155:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763672.861164:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763672.861182:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3647 00000100:00080000:2.0:1789763672.861200:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763672.861216:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763672.861285:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699513074560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763672.861327:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763672.861333:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763672.861336:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763672.861347:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3647 00000100:00080000:2.0:1789763672.861353:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763672.861362:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763672.861376:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763672.861406:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699513074688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763672.861412:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513074560:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763672.861446:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763672.861453:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763672.861458:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763672.861472:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3647 00000100:00080000:2.0:1789763672.861481:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763672.861488:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763672.861500:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00010000:00010000:1.0:1789763672.861502:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763672.861527:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699513074816/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763672.861544:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763672.861550:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763672.861553:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763672.861558:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3647 00000100:00080000:2.0:1789763672.861564:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763672.861574:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763672.861589:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1789763672.861626:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763672.861644:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699513074944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763672.861825:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763672/real 1789763672] req@ffff8dc00368df80 x1876699513074560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763672.861856:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699513074560/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763672.862165:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763672.862177:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763672.862186:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513074560:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763672.862238:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513074688:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763672.862376:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763672.862410:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763672/real 1789763672] req@ffff8dc00368f800 x1876699513074688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763672.862439:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699513074688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763672.862459:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763672.862466:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763672.862475:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513074688:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789763672.862489:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513074816:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789763672.862535:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763672.862556:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763672/real 1789763672] req@ffff8dc00368e300 x1876699513074816/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763672.862573:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699513074816/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763672.862584:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763672.862589:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763672.862626:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513074816:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789763672.862636:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513074944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763672.862669:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763672.862686:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763672/real 1789763672] req@ffff8dc00368ed80 x1876699513074944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763672.862702:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699513074944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763672.862713:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763672.862717:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763672.862721:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513074944:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763673.885140:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763674.909145:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763675.933211:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763676.957156:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763677.981040:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763677.981056:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763677.981063:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763677.981081:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3652 00000100:00080000:2.0:1789763677.981098:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763677.981112:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763677.981169:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763677.981182:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699513075072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763677.981226:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763677.981231:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763677.981233:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763677.981238:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3652 00000100:00080000:2.0:1789763677.981243:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763677.981249:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763677.981265:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763677.981284:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699513075200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763677.981297:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513075072:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763677.981305:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763677.981313:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763677.981316:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763677.981322:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3652 00000100:00080000:2.0:1789763677.981327:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763677.981331:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763677.981338:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763677.981354:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699513075328/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763677.981367:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763677.981372:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763677.981375:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763677.981381:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3652 00000100:00080000:2.0:1789763677.981399:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763677.981405:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763677.981412:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763677.981427:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699513075456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763677.981485:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513075200:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789763677.981527:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513075328:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789763677.981593:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513075456:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763679.005359:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763680.029276:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763681.053524:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763681.053594:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:2.0:1789763681.053674:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763681.053706:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789763681.053783:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789763681.053818:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789763681.053838:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:3.0:1789763681.053935:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763677/real 1789763681] req@ffff8dc00368ea00 x1876699513075200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763681.053991:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699513075200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763681.054016:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763681.054026:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763681.054046:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513075200:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789763681.054064:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763677/real 1789763681] req@ffff8dc00368c000 x1876699513075328/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763681.054094:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699513075328/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763681.054115:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763681.054123:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763681.054133:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513075328:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789763681.054149:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763677/real 1789763681] req@ffff8dc00368ce00 x1876699513075456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763681.054175:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699513075456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763681.054191:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763681.054198:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763681.054204:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513075456:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789763681.054219:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763677/real 1789763681] req@ffff8dc00368ca80 x1876699513075072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763681.054244:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699513075072/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763681.054261:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763681.054268:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763681.054275:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513075072:192.168.203.156@tcp:250:kworker.0 00010000:00010000:1.0:1789763682.077202:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763683.101004:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763683.101014:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763683.101019:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763683.101030:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3657 00000100:00080000:2.0:1789763683.101041:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763683.101055:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763683.101100:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699513075584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763683.101122:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763683.101125:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763683.101127:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763683.101131:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3657 00010000:00010000:3.0:1789763683.101133:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763683.101135:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763683.101142:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763683.101151:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763683.101162:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513075584:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763683.101169:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699513075712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763683.101181:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763683.101186:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763683.101188:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763683.101192:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3657 00000100:00080000:2.0:1789763683.101196:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763683.101201:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763683.101207:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763683.101221:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699513075840/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763683.101233:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763683.101237:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763683.101239:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763683.101244:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3657 00000100:00080000:2.0:1789763683.101249:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763683.101253:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763683.101259:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763683.101265:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763683.101277:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699513075968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763683.101328:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763683/real 1789763683] req@ffff8dc00368fb80 x1876699513075584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763683.101350:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699513075584/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763683.101363:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763683.101368:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763683.101374:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513075584:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763683.101408:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513075712:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763683.101434:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763683.101449:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763683/real 1789763683] req@ffff8dc00368f480 x1876699513075712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763683.101462:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699513075712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763683.101471:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763683.101474:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763683.101478:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513075712:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763683.101485:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513075840:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763683.101507:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763683.101536:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763683/real 1789763683] req@ffff8dc00368f100 x1876699513075840/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763683.101550:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699513075840/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763683.101564:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763683.101568:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763683.101572:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513075840:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763683.101578:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513075968:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763683.101599:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763683.101609:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763683/real 1789763683] req@ffff8dc00368d880 x1876699513075968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763683.101621:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699513075968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763683.101630:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763683.101633:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763683.101637:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513075968:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763684.125325:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763685.149149:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763686.173126:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763687.197177:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763688.221081:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763688.221098:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763688.221105:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763688.221123:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3662 00000100:00080000:1.0:1789763688.221139:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763688.221153:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789763688.221217:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763688.221237:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699513076096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763688.221287:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763688.221294:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763688.221297:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763688.221304:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3662 00000100:00080000:1.0:1789763688.221311:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763688.221318:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763688.221335:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763688.221363:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513076096:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763688.221364:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699513076224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763688.221407:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763688.221413:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763688.221416:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763688.221424:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3662 00000100:00080000:1.0:1789763688.221429:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763688.221435:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763688.221451:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763688.221470:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699513076352/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763688.221504:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789763688.221526:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763688.221532:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763688.221536:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763688.221545:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3662 00000100:00080000:1.0:1789763688.221551:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763688.221560:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763688.221570:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763688.221577:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763688/real 1789763688] req@ffff8dc005c94a80 x1876699513076096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763688.221597:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699513076480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763688.221619:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699513076096/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763688.221638:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763688.221648:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763688.221654:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513076096:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763688.221689:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513076224:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763688.221720:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763688.221737:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763688/real 1789763688] req@ffff8dc005c96a00 x1876699513076224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763688.221752:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699513076224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763688.221762:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763688.221767:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763688.221771:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513076224:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763688.221780:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513076352:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763688.221801:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763688.221813:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763688/real 1789763688] req@ffff8dc005c94700 x1876699513076352/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763688.221827:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699513076352/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763688.221837:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763688.221841:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763688.221845:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513076352:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763688.221852:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513076480:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763688.221871:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763688.221882:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763688/real 1789763688] req@ffff8dc005c95f80 x1876699513076480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763688.221899:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699513076480/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763688.221967:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763688.221976:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763688.221983:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513076480:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763689.245121:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763690.269184:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763691.293224:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763692.317097:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763693.341172:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:1.0:1789763693.341185:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763693.341186:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763693.341196:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763693.341214:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3667 00000100:00080000:2.0:1789763693.341234:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763693.341244:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763693.341296:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699513076608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763693.341337:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763693.341343:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763693.341345:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763693.341351:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3667 00000100:00080000:2.0:1789763693.341355:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763693.341359:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763693.341371:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763693.341388:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699513076736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763693.341405:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763693.341411:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763693.341415:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763693.341427:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3667 00000100:00100000:0.0:1789763693.341433:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513076608:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763693.341435:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763693.341478:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763693.341493:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763693.341525:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699513076864/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763693.341548:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763693.341556:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763693.341559:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763693.341568:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3667 00000100:00080000:2.0:1789763693.341578:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763693.341587:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789763693.341599:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:2.0:1789763693.341599:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763693.341635:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699513076992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763693.341701:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763693/real 1789763693] req@ffff8dc00368d880 x1876699513076608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763693.341732:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699513076608/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763693.341750:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763693.341756:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763693.341762:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513076608:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763693.341801:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513076736:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763693.341837:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763693.341854:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763693/real 1789763693] req@ffff8dc00368f100 x1876699513076736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763693.341896:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699513076736/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763693.341943:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763693.341948:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763693.341952:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513076736:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763693.341962:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513076864:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763693.341992:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763693.342009:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763693/real 1789763693] req@ffff8dc00368f480 x1876699513076864/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763693.342024:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699513076864/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763693.342043:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763693.342048:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763693.342052:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513076864:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763693.342059:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513076992:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763693.342081:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763693.342093:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763693/real 1789763693] req@ffff8dc00368fb80 x1876699513076992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763693.342107:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699513076992/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763693.342118:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763693.342122:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763693.342125:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513076992:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763694.365214:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763695.389158:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763696.413193:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763697.437169:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763698.461053:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763698.461069:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763698.461077:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763698.461094:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3673 00000100:00080000:2.0:1789763698.461111:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763698.461124:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763698.461140:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763698.461176:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699513077120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763698.461214:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763698.461221:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763698.461225:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763698.461234:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3673 00000100:00080000:2.0:1789763698.461241:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763698.461249:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763698.461265:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763698.461283:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513077120:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763698.461290:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699513077248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763698.461309:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763698.461315:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763698.461318:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763698.461325:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3673 00000100:00080000:2.0:1789763698.461332:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763698.461338:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763698.461347:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763698.461366:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699513077376/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763698.461384:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763698.461391:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1789763698.461395:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789763698.461395:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763698.461431:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3673 00000100:00080000:2.0:1789763698.461438:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763698.461445:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763698.461455:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763698.461474:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699513077504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763698.461484:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763698/real 1789763698] req@ffff8dc00368ca80 x1876699513077120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763698.461517:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699513077120/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763698.461538:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763698.461546:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763698.461554:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513077120:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763698.461590:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513077248:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763698.461626:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763698.461644:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763698/real 1789763698] req@ffff8dc00368ce00 x1876699513077248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763698.461660:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699513077248/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763698.461671:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763698.461675:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763698.461679:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513077248:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763698.461687:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513077376:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763698.461707:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763698.461719:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763698/real 1789763698] req@ffff8dc00368c000 x1876699513077376/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763698.461733:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699513077376/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763698.461743:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763698.461746:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763698.461750:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513077376:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763698.461757:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513077504:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763698.461776:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763698.461787:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763698/real 1789763698] req@ffff8dc00368ea00 x1876699513077504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763698.461801:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699513077504/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763698.461811:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763698.461828:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763698.461833:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513077504:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763699.485149:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763700.509175:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763701.533033:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763702.557080:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763703.581024:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763703.581033:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763703.581036:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1789763703.581047:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763703.581048:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3678 00000100:00080000:2.0:1789763703.581063:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763703.581070:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763703.581102:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699513077632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763703.581125:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763703.581129:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763703.581131:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763703.581136:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3678 00000100:00080000:2.0:1789763703.581142:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763703.581147:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763703.581158:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763703.581167:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513077632:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763703.581177:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699513077760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763703.581211:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763703.581214:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763703.581216:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763703.581222:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3678 00000100:00080000:2.0:1789763703.581226:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763703.581230:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763703.581238:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763703.581253:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699513077888/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763703.581264:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789763703.581265:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763703.581278:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763703.581280:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763703.581289:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3678 00000100:00080000:2.0:1789763703.581294:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763703.581299:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763703.581305:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763703.581315:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699513078016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763703.581328:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763703/real 1789763703] req@ffff8dc00368ed80 x1876699513077632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763703.581349:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699513077632/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763703.581360:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763703.581378:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763703.581382:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513077632:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763703.581425:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513077760:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763703.581444:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763703.581455:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763703/real 1789763703] req@ffff8dc00368e300 x1876699513077760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763703.581464:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699513077760/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763703.581476:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763703.581479:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763703.581482:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513077760:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763703.581487:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513077888:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763703.581500:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763703.581507:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763703/real 1789763703] req@ffff8dc00368f800 x1876699513077888/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763703.581516:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699513077888/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763703.581522:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763703.581524:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763703.581526:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513077888:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763703.581530:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513078016:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763703.581543:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763703.581550:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763703/real 1789763703] req@ffff8dc00368df80 x1876699513078016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763703.581558:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699513078016/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763703.581564:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763703.581566:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763703.581574:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513078016:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763704.605133:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763705.629095:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763706.653157:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763707.677181:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763708.701129:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763708.701143:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763708.701151:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789763708.701153:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763708.701171:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3683 00000100:00080000:1.0:1789763708.701188:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763708.701202:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763708.701254:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699513078144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763708.701298:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763708.701304:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763708.701307:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763708.701312:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3683 00000100:00080000:1.0:1789763708.701319:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763708.701325:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763708.701368:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763708.701390:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513078144:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763708.701396:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699513078272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763708.701426:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763708.701431:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763708.701435:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763708.701446:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3683 00000100:00080000:1.0:1789763708.701455:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763708.701464:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763708.701476:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763708.701497:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699513078400/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763708.701511:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763708.701517:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763708.701519:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763708.701528:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3683 00000800:00000100:2.0:1789763708.701534:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789763708.701539:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763708.701551:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763708.701561:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763708.701587:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699513078528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763708.701606:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763708/real 1789763708] req@ffff8dc005c94380 x1876699513078144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763708.701645:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699513078144/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763708.701661:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763708.701667:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763708.701674:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513078144:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763708.701707:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513078272:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763708.701742:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763708.701758:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763708/real 1789763708] req@ffff8dc005c97b80 x1876699513078272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763708.701774:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699513078272/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763708.701785:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763708.701789:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763708.701793:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513078272:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763708.701800:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513078400:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789763708.701820:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763708.701832:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763708/real 1789763708] req@ffff8dc005c96300 x1876699513078400/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763708.701849:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699513078400/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763708.701868:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763708.701872:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763708.701876:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513078400:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763708.701883:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513078528:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763708.701986:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763708.702016:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763708/real 1789763708] req@ffff8dc005c95500 x1876699513078528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763708.702038:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699513078528/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763708.702050:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763708.702055:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763708.702059:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513078528:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763709.725198:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763710.749213:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763711.773130:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763712.797283:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763713.821128:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00010000:00010000:2.0:1789763713.821144:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763713.821147:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763713.821158:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763713.821175:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3688 00000100:00080000:1.0:1789763713.821190:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763713.821199:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763713.821269:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699513078656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763713.821345:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763713.821353:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763713.821366:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763713.821374:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3688 00000100:00080000:1.0:1789763713.821380:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1789763713.821387:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513078656:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789763713.821407:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763713.821437:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763713.821470:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699513078784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763713.821490:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763713.821496:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763713.821499:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763713.821505:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3688 00000100:00080000:1.0:1789763713.821514:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763713.821519:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763713.821526:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763713.821543:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699513078912/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763713.821574:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1789763713.821574:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789763713.821585:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763713.821587:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763713.821596:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3688 00000100:00080000:1.0:1789763713.821601:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763713.821607:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763713.821615:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763713.821628:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699513079040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763713.821650:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763713/real 1789763713] req@ffff8dc005c95880 x1876699513078656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763713.821674:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699513078656/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763713.821687:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763713.821692:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763713.821696:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513078656:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763713.821725:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513078784:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763713.821746:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763713.821756:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763713/real 1789763713] req@ffff8dc005c94e00 x1876699513078784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763713.821766:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699513078784/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763713.821773:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763713.821776:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763713.821778:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513078784:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763713.821783:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513078912:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789763713.821797:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763713.821804:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763713/real 1789763713] req@ffff8dc005c97480 x1876699513078912/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763713.821813:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699513078912/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763713.821819:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763713.821821:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763713.821823:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513078912:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763713.821828:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513079040:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763713.821841:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763713.821849:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763713/real 1789763713] req@ffff8dc005c96680 x1876699513079040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763713.821857:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699513079040/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763713.821864:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763713.821868:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763713.821872:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513079040:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763714.845070:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763715.869202:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763716.893113:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763717.917109:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763718.941080:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763718.941103:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763718.941109:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763718.941123:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3693 00000100:00080000:3.0:1789763718.941135:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763718.941145:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789763718.941168:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763718.941196:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699513079168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763718.941234:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763718.941239:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763718.941242:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763718.941247:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3693 00000100:00080000:3.0:1789763718.941280:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763718.941288:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763718.941301:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763718.941326:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513079168:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763718.941327:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699513079296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763718.941350:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763718.941362:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763718.941365:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763718.941371:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3693 00000100:00080000:3.0:1789763718.941377:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763718.941382:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763718.941389:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763718.941407:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699513079424/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763718.941421:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763718.941425:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763718.941428:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763718.941433:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3693 00000100:00080000:3.0:1789763718.941438:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763718.941444:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:2.0:1789763718.941454:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:3.0:1789763718.941456:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763718.941476:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699513079552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763718.941519:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763718/real 1789763718] req@ffff8dc0107ac700 x1876699513079168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763718.941545:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699513079168/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763718.941560:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763718.941565:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763718.941570:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513079168:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763718.941601:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513079296:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763718.941629:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763718.941645:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763718/real 1789763718] req@ffff8dc0107aed80 x1876699513079296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763718.941669:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699513079296/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763718.941680:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763718.941684:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763718.941688:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513079296:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763718.941695:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513079424:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789763718.941716:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763718.941727:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763718/real 1789763718] req@ffff8dc0107adf80 x1876699513079424/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763718.941741:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699513079424/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763718.941751:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763718.941755:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763718.941758:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513079424:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763718.941765:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513079552:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763718.941785:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763718.941796:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763718/real 1789763718] req@ffff8dc0107ac380 x1876699513079552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763718.941812:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699513079552/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763718.941842:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763718.941848:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763718.941855:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513079552:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763719.965158:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763720.989013:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763722.013195:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763723.037217:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763724.061097:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763724.061113:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763724.061121:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763724.061139:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3698 00010000:00010000:0.0:1789763724.061161:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763724.061165:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763724.061182:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763724.061277:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699513079680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763724.061321:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763724.061330:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763724.061333:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763724.061339:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3698 00000100:00080000:3.0:1789763724.061345:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763724.061352:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763724.061367:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763724.061386:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699513079808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763724.061398:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513079680:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763724.061424:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763724.061431:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763724.061433:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763724.061439:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3698 00000100:00080000:3.0:1789763724.061445:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763724.061452:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763724.061460:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763724.061481:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699513079936/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763724.061493:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763724.061498:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763724.061501:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763724.061506:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3698 00000100:00080000:3.0:1789763724.061517:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789763724.061525:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763724.061539:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763724.061553:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763724.061593:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763724/real 1789763724] req@ffff8dc0107af800 x1876699513079680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763724.061597:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699513080064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763724.061669:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699513079680/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763724.061689:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763724.061696:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763724.061707:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513079680:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763724.061749:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513079808:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763724.061778:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763724.061794:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763724/real 1789763724] req@ffff8dc0107ae680 x1876699513079808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763724.061809:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699513079808/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763724.061820:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763724.061825:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763724.061829:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513079808:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763724.061838:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513079936:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789763724.061858:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763724.061870:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763724/real 1789763724] req@ffff8dc0107ace00 x1876699513079936/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763724.061883:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699513079936/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763724.061894:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763724.061934:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763724.061939:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513079936:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763724.061947:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513080064:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763724.061973:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763724.061987:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763724/real 1789763724] req@ffff8dc0107af100 x1876699513080064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763724.062002:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699513080064/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763724.062012:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763724.062016:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763724.062020:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513080064:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763725.085122:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763726.109102:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763727.133119:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763728.157183:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763729.181087:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763729.181100:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763729.181106:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763729.181120:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3703 00000100:00080000:3.0:1789763729.181137:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763729.181155:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763729.181229:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763729.181244:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699513080192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763729.181289:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763729.181294:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763729.181297:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763729.181303:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3703 00000100:00080000:3.0:1789763729.181310:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763729.181317:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763729.181331:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763729.181350:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513080192:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763729.181355:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699513080320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763729.181376:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763729.181380:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763729.181383:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763729.181388:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3703 00000100:00080000:3.0:1789763729.181394:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763729.181399:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763729.181407:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763729.181423:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699513080448/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763729.181438:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763729.181442:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763729.181450:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1789763729.181451:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763729.181466:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3703 00000100:00080000:3.0:1789763729.181473:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763729.181478:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763729.181485:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763729.181502:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699513080576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763729.181518:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763729/real 1789763729] req@ffff8dc0107ad880 x1876699513080192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763729.181547:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699513080192/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763729.181562:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763729.181569:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763729.181575:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513080192:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763729.181605:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513080320:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763729.181632:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763729.181648:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763729/real 1789763729] req@ffff8dc0107aea00 x1876699513080320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763729.181663:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699513080320/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763729.181674:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763729.181678:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763729.181682:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513080320:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763729.181691:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513080448:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789763729.181726:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763729.181749:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763729/real 1789763729] req@ffff8dc0107af100 x1876699513080448/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763729.181789:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699513080448/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763729.181806:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763729.181811:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763729.181816:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513080448:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763729.181824:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513080576:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763729.181853:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763729.181869:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763729/real 1789763729] req@ffff8dc0107ace00 x1876699513080576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763729.181893:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699513080576/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763729.181968:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763729.181975:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763729.181982:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513080576:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763730.205162:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763731.229235:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763732.253200:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763733.277118:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763734.301081:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763734.301104:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763734.301112:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763734.301136:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3708 00000100:00080000:3.0:1789763734.301182:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763734.301198:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763734.301230:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763734.301270:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699513080704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763734.301332:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763734.301341:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763734.301344:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763734.301354:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3708 00000100:00080000:3.0:1789763734.301364:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763734.301374:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763734.301393:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763734.301412:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513080704:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763734.301430:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699513080832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763734.301460:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763734.301468:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763734.301473:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763734.301481:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3708 00000100:00080000:3.0:1789763734.301507:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789763734.301520:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763734.301523:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763734.301541:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763734.301578:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699513080960/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763734.301601:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1789763734.301612:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763734/real 1789763734] req@ffff8dc0107ae680 x1876699513080704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763734.301615:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763734.301624:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763734.301646:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3708 00000100:00080000:3.0:1789763734.301663:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763734.301677:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763734.301680:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699513080704/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1789763734.301693:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763734.301725:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699513081088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763734.301728:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763734.301746:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763734.301753:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513080704:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763734.301799:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513080832:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763734.301846:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763734.301869:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763734/real 1789763734] req@ffff8dc0107af800 x1876699513080832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763734.301886:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699513080832/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763734.301898:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763734.301980:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763734.301988:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513080832:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763734.302004:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513080960:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789763734.302043:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763734.302060:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763734/real 1789763734] req@ffff8dc0107ac380 x1876699513080960/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763734.302076:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699513080960/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763734.302087:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763734.302092:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763734.302096:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513080960:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763734.302104:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513081088:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763734.302126:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763734.302168:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763734/real 1789763734] req@ffff8dc0107adf80 x1876699513081088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763734.302190:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699513081088/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763734.302207:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763734.302214:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763734.302221:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513081088:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763735.325291:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763736.349216:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763737.373184:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763738.397198:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763739.421088:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763739.421130:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763739.421137:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763739.421163:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3714 00000100:00080000:2.0:1789763739.421179:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763739.421190:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789763739.421246:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763739.421256:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699513081216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763739.421299:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763739.421305:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763739.421307:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763739.421313:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3714 00000100:00080000:2.0:1789763739.421318:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763739.421325:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763739.421348:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763739.421372:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699513081344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763739.421375:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513081216:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763739.421395:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763739.421402:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763739.421404:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763739.421409:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3714 00000100:00080000:2.0:1789763739.421414:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763739.421418:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763739.421425:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763739.421439:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699513081472/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763739.421453:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763739.421458:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763739.421460:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763739.421466:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3714 00000100:00080000:2.0:1789763739.421471:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763739.421476:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763739.421481:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763739.421494:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699513081600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763739.421510:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763739.421577:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763739/real 1789763739] req@ffff8dc00368dc00 x1876699513081216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763739.421603:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699513081216/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763739.421619:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763739.421625:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763739.421630:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513081216:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763739.421666:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513081344:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763739.421704:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763739.421730:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763739/real 1789763739] req@ffff8dc00368e680 x1876699513081344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763739.421758:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699513081344/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763739.421776:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763739.421784:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763739.421789:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513081344:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763739.421798:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513081472:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763739.421830:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763739.421846:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763739/real 1789763739] req@ffff8dc00368d180 x1876699513081472/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763739.421861:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699513081472/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763739.421872:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763739.421876:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763739.421880:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513081472:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763739.421886:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513081600:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763739.421943:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763739.421957:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763739/real 1789763739] req@ffff8dc00368c700 x1876699513081600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763739.421973:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699513081600/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763739.421983:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763739.421987:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763739.421991:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513081600:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763740.445200:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763741.469144:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763742.493236:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763743.517212:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763744.541088:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763744.541122:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763744.541130:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763744.541145:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3719 00010000:00010000:1.0:1789763744.541150:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763744.541163:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763744.541178:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763744.541224:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699513081728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763744.541273:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763744.541278:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763744.541281:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763744.541287:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3719 00000100:00080000:2.0:1789763744.541294:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763744.541300:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763744.541312:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763744.541339:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513081728:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763744.541340:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699513081856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763744.541375:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763744.541383:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763744.541403:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763744.541410:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3719 00000100:00080000:2.0:1789763744.541416:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763744.541424:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763744.541433:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763744.541454:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699513081984/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763744.541468:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763744.541472:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763744.541474:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763744.541479:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3719 00000100:00080000:2.0:1789763744.541485:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763744.541490:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763744.541497:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763744.541513:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513081856:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763744.541515:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699513082112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763744.541565:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513081984:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763744.541598:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513082112:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789763745.565235:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763746.589237:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763747.613174:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1789763747.613397:0:2346:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:3.0:1789763747.613468:0:2346:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1789763747.613494:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789763747.613547:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789763747.613559:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:3.0:1789763747.613579:0:2346:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:0.0:1789763747.613631:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763744/real 1789763747] req@ffff8dc00368dc00 x1876699513082112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763747.613688:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699513082112/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763747.613715:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763747.613725:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763747.613735:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513082112:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763747.613753:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763744/real 1789763747] req@ffff8dc00368d180 x1876699513081856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763747.613782:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699513081856/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763747.613811:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763747.613821:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763747.613828:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513081856:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763747.613843:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763744/real 1789763747] req@ffff8dc00368e680 x1876699513081984/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763747.613886:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699513081984/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763747.613955:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763747.613965:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763747.613975:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513081984:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763747.613992:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763744/real 1789763747] req@ffff8dc00368c700 x1876699513081728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763747.614022:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699513081728/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763747.614068:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763747.614078:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763747.614086:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513081728:192.168.203.156@tcp:250:kworker.0 00010000:00010000:3.0:1789763748.637207:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763749.661186:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763749.661206:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763749.661212:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1789763749.661232:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763749.661233:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3724 00000100:00080000:3.0:1789763749.661255:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763749.661269:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763749.661330:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699513082240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763749.661406:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763749.661413:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763749.661417:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763749.661424:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3724 00000100:00080000:3.0:1789763749.661432:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763749.661449:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763749.661470:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763749.661498:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513082240:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763749.661504:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699513082368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763749.661542:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763749.661550:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763749.661554:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763749.661562:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3724 00000100:00080000:3.0:1789763749.661572:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763749.661581:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763749.661593:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763749.661619:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699513082496/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763749.661644:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763749.661651:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763749.661661:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763749.661665:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763749.661676:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3724 00000100:00080000:3.0:1789763749.661689:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763749.661698:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763749.661708:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763749.661734:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763749/real 1789763749] req@ffff8dc0107aed80 x1876699513082240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763749.661743:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699513082624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763749.661882:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699513082240/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763749.661957:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763749.661968:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763749.661976:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513082240:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763749.662066:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513082368:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763749.662127:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763749.662156:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763749/real 1789763749] req@ffff8dc0107ac700 x1876699513082368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763749.662184:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699513082368/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763749.662204:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763749.662211:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763749.662218:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513082368:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763749.662234:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513082496:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763749.662278:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763749.662316:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763749/real 1789763749] req@ffff8dc0107ad180 x1876699513082496/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763749.662341:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699513082496/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763749.662357:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763749.662363:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763749.662370:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513082496:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763749.662385:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513082624:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763749.662429:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763749.662457:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763749/real 1789763749] req@ffff8dc0107af480 x1876699513082624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763749.662498:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699513082624/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763749.662515:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763749.662521:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763749.662527:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513082624:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763750.685154:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763751.709248:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763752.733179:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763753.757180:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763754.781119:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763754.781136:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763754.781143:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763754.781163:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3729 00000100:00080000:1.0:1789763754.781183:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:2.0:1789763754.781193:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763754.781196:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763754.781271:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96d80 x1876699513082752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763754.781322:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763754.781330:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763754.781334:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763754.781344:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3729 00000100:00080000:1.0:1789763754.781353:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763754.781362:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763754.781383:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763754.781392:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513082752:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763754.781427:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699513082880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763754.781455:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763754.781460:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763754.781463:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763754.781468:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3729 00000100:00080000:1.0:1789763754.781494:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763754.781500:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789763754.781510:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1789763754.781515:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763754.781541:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699513083008/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763754.781558:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763754.781563:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763754.781566:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763754.781575:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3729 00000100:00080000:1.0:1789763754.781580:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763754.781586:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763754.781595:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763754.781600:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763754/real 1789763754] req@ffff8dc005c96d80 x1876699513082752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763754.781628:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699513083136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763754.781634:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96d80 x1876699513082752/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763754.781727:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763754.781736:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763754.781744:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513082752:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763754.781784:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513082880:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763754.781816:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763754.781833:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763754/real 1789763754] req@ffff8dc005c96680 x1876699513082880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763754.781849:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699513082880/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763754.781869:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763754.781874:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763754.781878:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513082880:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763754.781886:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513083008:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763754.781946:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763754.781961:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763754/real 1789763754] req@ffff8dc005c97480 x1876699513083008/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763754.781976:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699513083008/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763754.782011:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763754.782016:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763754.782021:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513083008:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763754.782030:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513083136:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763754.782056:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763754.782071:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763754/real 1789763754] req@ffff8dc005c94e00 x1876699513083136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763754.782086:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699513083136/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763754.782096:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763754.782100:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763754.782105:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513083136:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763755.805153:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763756.829169:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763757.853164:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763758.877176:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763759.901152:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763759.901170:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763759.901179:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1789763759.901182:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763759.901193:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3734 00000100:00080000:2.0:1789763759.901210:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763759.901247:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763759.901292:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699513083264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763759.901323:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763759.901327:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763759.901330:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763759.901335:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3734 00000100:00080000:2.0:1789763759.901340:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763759.901347:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763759.901361:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763759.901373:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513083264:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763759.901387:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699513083392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763759.901408:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763759.901415:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763759.901418:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763759.901426:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3734 00000100:00080000:2.0:1789763759.901433:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763759.901439:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763759.901446:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789763759.901454:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763759.901466:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699513083520/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763759.901523:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763759.901533:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1789763759.901538:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763759/real 1789763759] req@ffff8dc00368df80 x1876699513083264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763759.901540:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763759.901557:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3734 00000100:00080000:2.0:1789763759.901573:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763759.901586:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763759.901595:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763759.901603:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699513083264/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763759.901622:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699513083648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763759.901641:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763759.901665:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763759.901676:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513083264:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763759.901715:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513083392:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763759.901751:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763759.901768:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763759/real 1789763759] req@ffff8dc00368f800 x1876699513083392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763759.901801:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699513083392/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763759.901811:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763759.901814:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763759.901818:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513083392:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763759.901825:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513083520:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763759.901844:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763759.901856:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763759/real 1789763759] req@ffff8dc00368e300 x1876699513083520/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763759.901868:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699513083520/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763759.901876:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763759.901879:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763759.901882:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513083520:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763759.901888:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513083648:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763759.901903:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763759.901912:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763759/real 1789763759] req@ffff8dc00368ed80 x1876699513083648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763759.901923:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699513083648/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763759.901931:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763759.901982:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763759.901986:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513083648:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763760.925178:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763761.949082:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763762.973171:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763763.997238:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763765.021092:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763765.021106:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763765.021112:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763765.021130:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3739 00000100:00080000:2.0:1789763765.021148:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763765.021159:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763765.021222:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699513083776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1789763765.021254:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763765.021256:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763765.021266:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763765.021269:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763765.021278:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3739 00000100:00080000:2.0:1789763765.021283:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763765.021290:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763765.021303:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763765.021321:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699513083904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763765.021330:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513083776:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763765.021339:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763765.021356:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763765.021361:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763765.021368:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3739 00000100:00080000:2.0:1789763765.021372:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763765.021376:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763765.021383:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763765.021404:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699513084032/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763765.021416:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763765.021421:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763765.021424:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763765.021430:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3739 00000100:00080000:2.0:1789763765.021434:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763765.021442:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763765.021448:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763765.021461:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699513084160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763765.021487:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763765.021571:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763765/real 1789763765] req@ffff8dc00368ea00 x1876699513083776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763765.021598:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699513083776/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763765.021616:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763765.021622:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763765.021628:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513083776:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763765.021667:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513083904:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763765.021695:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763765.021711:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763765/real 1789763765] req@ffff8dc00368c000 x1876699513083904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763765.021727:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699513083904/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763765.021738:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763765.021742:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763765.021746:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513083904:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763765.021754:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513084032:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763765.021775:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763765.021787:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763765/real 1789763765] req@ffff8dc00368ce00 x1876699513084032/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763765.021801:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699513084032/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763765.021812:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763765.021816:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763765.021819:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513084032:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763765.021827:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513084160:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763765.021848:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763765.021860:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763765/real 1789763765] req@ffff8dc00368ca80 x1876699513084160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763765.021875:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699513084160/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763765.021884:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763765.021888:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763765.021892:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513084160:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763766.045164:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763767.069118:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763768.093174:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763769.117023:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763770.141046:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763770.141056:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763770.141076:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763770.141086:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3744 00000100:00080000:2.0:1789763770.141097:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00010000:00010000:3.0:1789763770.141098:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763770.141105:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763770.141157:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699513084288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763770.141179:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763770.141183:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763770.141184:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763770.141188:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3744 00000100:00080000:2.0:1789763770.141191:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763770.141211:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763770.141219:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763770.141235:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513084288:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763770.141236:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699513084416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763770.141257:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763770.141276:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763770.141279:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763770.141286:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3744 00000100:00080000:2.0:1789763770.141293:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763770.141298:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763770.141304:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763770.141319:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699513084544/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763770.141334:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789763770.141336:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763770.141342:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763770.141345:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763770.141352:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3744 00000100:00080000:2.0:1789763770.141359:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763770.141364:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763770.141370:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763770.141382:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699513084672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763770.141390:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763770/real 1789763770] req@ffff8dc00368fb80 x1876699513084288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763770.141410:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699513084288/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763770.141420:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763770.141423:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763770.141426:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513084288:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763770.141447:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513084416:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763770.141485:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763770.141494:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763770/real 1789763770] req@ffff8dc00368f480 x1876699513084416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763770.141503:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699513084416/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763770.141508:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763770.141510:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763770.141512:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513084416:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763770.141516:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513084544:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763770.141528:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763770.141534:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763770/real 1789763770] req@ffff8dc00368f100 x1876699513084544/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763770.141541:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699513084544/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763770.141546:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763770.141548:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763770.141550:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513084544:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763770.141554:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513084672:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763770.141584:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763770.141591:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763770/real 1789763770] req@ffff8dc00368d880 x1876699513084672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763770.141605:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699513084672/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763770.141611:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763770.141613:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763770.141615:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513084672:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763771.165225:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763772.189065:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763773.213064:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763774.237163:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763775.261110:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763775.261137:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763775.261145:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763775.261163:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3749 00000100:00080000:1.0:1789763775.261182:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763775.261194:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789763775.261237:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763775.261258:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699513084800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763775.261308:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763775.261316:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763775.261320:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763775.261341:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3749 00000100:00080000:1.0:1789763775.261348:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763775.261355:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763775.261368:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763775.261382:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513084800:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763775.261393:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699513084928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763775.261411:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763775.261415:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763775.261418:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763775.261424:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3749 00000100:00080000:1.0:1789763775.261429:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763775.261433:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763775.261441:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763775.261455:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699513085056/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763775.261470:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763775.261475:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763775.261477:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763775.261481:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3749 00000100:00080000:1.0:1789763775.261486:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1789763775.261495:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789763775.261496:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763775.261509:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763775.261523:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699513085184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763775.261576:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763775/real 1789763775] req@ffff8dc005c95880 x1876699513084800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763775.261612:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699513084800/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763775.261627:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763775.261633:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763775.261650:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513084800:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763775.261680:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513084928:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763775.261712:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763775.261729:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763775/real 1789763775] req@ffff8dc005c95500 x1876699513084928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763775.261745:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699513084928/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763775.261764:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763775.261768:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763775.261773:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513084928:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763775.261780:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513085056:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763775.261801:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763775.261814:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763775/real 1789763775] req@ffff8dc005c96300 x1876699513085056/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763775.261828:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699513085056/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763775.261879:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763775.261886:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763775.261894:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513085056:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763775.261937:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513085184:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763775.261969:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763775.261985:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763775/real 1789763775] req@ffff8dc005c97b80 x1876699513085184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763775.262000:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699513085184/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763775.262010:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763775.262014:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763775.262019:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513085184:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763776.285136:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763777.309098:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763778.333190:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763779.357023:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763780.380977:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763780.380989:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763780.380993:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763780.381004:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3754 00000100:00080000:1.0:1789763780.381015:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763780.381023:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789763780.381023:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763780.381055:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699513085312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763780.381080:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763780.381083:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763780.381084:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763780.381087:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3754 00000100:00080000:1.0:1789763780.381090:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763780.381093:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763780.381101:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763780.381114:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513085312:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763780.381114:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699513085440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763780.381128:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763780.381131:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763780.381132:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763780.381135:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3754 00000100:00080000:1.0:1789763780.381138:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763780.381140:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763780.381143:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763780.381153:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699513085568/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763780.381160:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763780.381162:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763780.381164:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763780.381166:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3754 00000100:00080000:1.0:1789763780.381171:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763780.381175:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763780.381178:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763780.381188:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763780.381189:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699513085696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763780.381234:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763780/real 1789763780] req@ffff8dc005c94380 x1876699513085312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763780.381254:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699513085312/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763780.381265:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763780.381270:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763780.381275:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513085312:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763780.381300:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513085440:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763780.381330:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763780.381350:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763780/real 1789763780] req@ffff8dc005c95f80 x1876699513085440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763780.381366:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699513085440/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763780.381373:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763780.381376:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763780.381381:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513085440:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763780.381387:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513085568:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763780.381404:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763780.381413:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763780/real 1789763780] req@ffff8dc005c94700 x1876699513085568/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763780.381422:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699513085568/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763780.381427:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763780.381429:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763780.381431:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513085568:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763780.381435:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513085696:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763780.381447:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763780.381453:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763780/real 1789763780] req@ffff8dc005c96a00 x1876699513085696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763780.381461:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699513085696/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763780.381467:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763780.381469:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763780.381471:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513085696:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763781.405072:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763782.429199:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763783.453009:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763784.477086:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763785.501069:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763785.501081:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763785.501087:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1789763785.501104:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763785.501104:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3760 00000100:00080000:2.0:1789763785.501121:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763785.501134:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763785.501182:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699513085824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763785.501234:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763785.501237:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763785.501239:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763785.501243:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3760 00000100:00080000:2.0:1789763785.501248:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763785.501253:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763785.501265:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763785.501285:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699513085952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763785.501296:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513085824:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763785.501310:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763785.501313:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763785.501314:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763785.501318:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3760 00000100:00080000:2.0:1789763785.501321:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763785.501323:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763785.501328:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763785.501337:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699513086080/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763785.501344:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763785.501347:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763785.501359:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763785.501361:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3760 00000100:00080000:2.0:1789763785.501364:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763785.501366:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763785.501370:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763785.501376:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699513086208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763785.501397:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763785.501441:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763785/real 1789763785] req@ffff8dc00368d880 x1876699513085824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763785.501456:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699513085824/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763785.501466:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763785.501469:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763785.501473:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513085824:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763785.501497:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513085952:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763785.501514:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763785.501523:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763785/real 1789763785] req@ffff8dc00368f100 x1876699513085952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763785.501531:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699513085952/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763785.501537:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763785.501540:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763785.501542:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513085952:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763785.501545:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513086080:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763785.501558:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763785.501564:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763785/real 1789763785] req@ffff8dc00368f480 x1876699513086080/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763785.501571:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699513086080/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763785.501577:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763785.501579:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763785.501581:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513086080:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763785.501585:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513086208:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763785.501596:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763785.501602:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763785/real 1789763785] req@ffff8dc00368fb80 x1876699513086208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763785.501610:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699513086208/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763785.501615:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763785.501616:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763785.501618:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513086208:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763786.525074:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763787.549035:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763788.573043:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763789.597066:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763790.620973:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763790.620981:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763790.620985:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763790.620994:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3765 00000100:00080000:2.0:1789763790.621003:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763790.621009:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763790.621037:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763790.621043:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699513086336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763790.621065:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763790.621068:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763790.621069:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763790.621072:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3765 00000100:00080000:2.0:1789763790.621074:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763790.621077:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763790.621085:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763790.621098:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513086336:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763790.621098:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699513086464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763790.621112:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763790.621114:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763790.621116:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763790.621118:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3765 00000100:00080000:2.0:1789763790.621121:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763790.621123:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763790.621127:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763790.621135:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699513086592/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763790.621146:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763790.621149:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763790.621150:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763790.621153:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3765 00000100:00080000:2.0:1789763790.621156:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763790.621158:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763790.621161:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763790.621164:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763790.621171:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699513086720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763790.621202:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763790/real 1789763790] req@ffff8dc00368ca80 x1876699513086336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763790.621216:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699513086336/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763790.621224:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763790.621227:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763790.621230:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513086336:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763790.621247:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513086464:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763790.621261:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763790.621269:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763790/real 1789763790] req@ffff8dc00368ce00 x1876699513086464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763790.621278:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699513086464/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763790.621283:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763790.621285:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763790.621287:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513086464:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763790.621291:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513086592:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763790.621302:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763790.621308:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763790/real 1789763790] req@ffff8dc00368c000 x1876699513086592/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763790.621315:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699513086592/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763790.621320:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763790.621323:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763790.621324:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513086592:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763790.621328:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513086720:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763790.621338:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763790.621343:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763790/real 1789763790] req@ffff8dc00368ea00 x1876699513086720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763790.621350:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699513086720/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763790.621359:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763790.621361:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763790.621363:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513086720:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763791.645046:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763792.669036:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763793.693012:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763794.717126:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763795.740989:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763795.740997:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763795.741001:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763795.741009:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3770 00000100:00080000:2.0:1789763795.741018:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763795.741024:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763795.741051:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763795.741065:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699513086848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763795.741086:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763795.741090:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763795.741092:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763795.741096:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3770 00000100:00080000:2.0:1789763795.741101:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763795.741106:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763795.741116:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763795.741126:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513086848:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763795.741133:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699513086976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763795.741143:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763795.741146:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763795.741147:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763795.741150:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3770 00000100:00080000:2.0:1789763795.741153:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763795.741155:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763795.741159:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763795.741168:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699513087104/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763795.741176:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763795.741178:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763795.741179:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763795.741182:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3770 00000100:00080000:2.0:1789763795.741186:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763795.741188:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763795.741191:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763795.741198:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763795.741200:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699513087232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763795.741242:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763795/real 1789763795] req@ffff8dc00368ed80 x1876699513086848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763795.741262:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699513086848/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763795.741273:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763795.741277:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763795.741279:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513086848:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763795.741299:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513086976:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763795.741317:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763795.741326:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763795/real 1789763795] req@ffff8dc00368e300 x1876699513086976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763795.741334:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699513086976/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763795.741340:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763795.741342:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763795.741344:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513086976:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763795.741348:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513087104:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763795.741359:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763795.741365:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763795/real 1789763795] req@ffff8dc00368f800 x1876699513087104/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763795.741383:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699513087104/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763795.741391:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763795.741393:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763795.741395:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513087104:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763795.741399:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513087232:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763795.741411:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763795.741417:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763795/real 1789763795] req@ffff8dc00368df80 x1876699513087232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763795.741425:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699513087232/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763795.741430:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763795.741431:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763795.741433:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513087232:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763796.765041:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763797.789025:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763798.813035:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763799.837092:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763800.861035:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763800.861045:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763800.861050:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763800.861062:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3775 00000100:00080000:2.0:1789763800.861072:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763800.861079:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763800.861103:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763800.861122:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699513087360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763800.861147:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763800.861150:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763800.861151:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763800.861155:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3775 00000100:00080000:2.0:1789763800.861157:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763800.861161:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763800.861170:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763800.861180:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699513087488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763800.861188:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763800.861192:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513087360:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763800.861192:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763800.861196:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763800.861204:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3775 00000100:00080000:2.0:1789763800.861207:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763800.861209:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763800.861212:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763800.861222:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699513087616/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763800.861234:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763800.861237:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763800.861239:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763800.861242:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3775 00000100:00080000:2.0:1789763800.861244:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763800.861247:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763800.861249:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763800.861256:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699513087744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763800.861277:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763800.861331:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763800/real 1789763800] req@ffff8dc00368c700 x1876699513087360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763800.861350:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699513087360/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763800.861360:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763800.861363:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763800.861367:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513087360:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763800.861390:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513087488:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763800.861407:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763800.861416:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763800/real 1789763800] req@ffff8dc00368e680 x1876699513087488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763800.861425:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699513087488/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763800.861431:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763800.861433:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763800.861435:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513087488:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763800.861439:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513087616:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763800.861450:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763800.861456:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763800/real 1789763800] req@ffff8dc00368d180 x1876699513087616/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763800.861464:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699513087616/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763800.861474:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763800.861476:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763800.861478:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513087616:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763800.861482:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513087744:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763800.861492:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763800.861498:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763800/real 1789763800] req@ffff8dc00368dc00 x1876699513087744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763800.861505:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699513087744/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763800.861510:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763800.861512:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763800.861514:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513087744:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763801.885088:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763802.909043:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763803.933068:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763804.957086:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763805.981004:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763805.981012:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763805.981015:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763805.981024:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3780 00000100:00080000:3.0:1789763805.981032:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763805.981038:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763805.981068:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763805.981070:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699513087872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763805.981091:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763805.981095:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763805.981097:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763805.981101:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3780 00000100:00080000:3.0:1789763805.981105:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763805.981110:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763805.981121:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763805.981125:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513087872:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763805.981138:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699513088000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763805.981152:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763805.981154:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763805.981156:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763805.981159:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3780 00000100:00080000:3.0:1789763805.981163:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763805.981166:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763805.981170:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789763805.981182:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763805.981183:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699513088128/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763805.981192:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763805.981195:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763805.981196:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763805.981200:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3780 00000100:00080000:3.0:1789763805.981202:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763805.981204:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763805.981207:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763805.981214:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699513088256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763805.981237:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763805/real 1789763805] req@ffff8dc0107ad500 x1876699513087872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763805.981259:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699513087872/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763805.981269:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763805.981272:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763805.981276:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513087872:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763805.981295:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513088000:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763805.981313:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763805.981322:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763805/real 1789763805] req@ffff8dc0107ac000 x1876699513088000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763805.981331:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699513088000/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763805.981336:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763805.981339:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763805.981341:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513088000:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763805.981345:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513088128:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763805.981356:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763805.981362:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763805/real 1789763805] req@ffff8dc0107afb80 x1876699513088128/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763805.981370:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699513088128/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763805.981375:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763805.981377:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763805.981378:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513088128:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763805.981382:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513088256:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763805.981392:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763805.981398:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763805/real 1789763805] req@ffff8dc0107ae300 x1876699513088256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763805.981405:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699513088256/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763805.981410:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763805.981412:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763805.981414:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513088256:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763807.005039:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763808.029046:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763809.053048:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763810.077136:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763811.101225:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763811.101246:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763811.101264:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763811.101285:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3785 00010000:00010000:1.0:1789763811.101297:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763811.101305:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763811.101322:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763811.101415:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699513088384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763811.101466:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763811.101472:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763811.101474:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763811.101480:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3785 00000100:00080000:2.0:1789763811.101486:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763811.101494:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763811.101538:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763811.101558:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513088384:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763811.101563:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699513088512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763811.101640:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763811.101644:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763811.101647:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763811.101653:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3785 00000100:00080000:2.0:1789763811.101658:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763811.101664:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763811.101672:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763811.101689:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699513088640/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763811.101703:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763811.101708:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763811.101710:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763811.101714:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3785 00000100:00080000:2.0:1789763811.101719:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763811.101723:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763811.101729:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763811.101748:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699513088768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763811.101779:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513088512:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763811.101847:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513088640:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763811.101877:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513088768:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763812.125188:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763813.149081:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763814.173168:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789763814.173255:0:2347:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1789763814.173304:0:2347:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789763814.173331:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789763814.173399:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789763814.173409:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789763814.173420:0:2347:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1789763814.173449:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763811/real 1789763814] req@ffff8dc00368d180 x1876699513088512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763814.173479:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699513088512/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763814.173492:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763814.173497:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763814.173502:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513088512:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763814.173510:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763811/real 1789763814] req@ffff8dc00368e680 x1876699513088640/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763814.173522:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699513088640/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763814.173529:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763814.173533:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763814.173536:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513088640:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763814.173549:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763811/real 1789763814] req@ffff8dc00368c700 x1876699513088768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763814.173575:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699513088768/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763814.173582:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763814.173586:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763814.173590:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513088768:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763814.173597:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763811/real 1789763814] req@ffff8dc00368dc00 x1876699513088384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763814.173615:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699513088384/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763814.173622:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763814.173625:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763814.173628:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513088384:192.168.203.156@tcp:250:kworker.0 00010000:00010000:1.0:1789763815.197095:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763816.221038:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763816.221052:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763816.221059:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763816.221075:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3790 00000100:00080000:2.0:1789763816.221093:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763816.221106:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789763816.221129:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763816.221189:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699513088896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763816.221236:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763816.221242:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763816.221245:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763816.221251:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3790 00000100:00080000:2.0:1789763816.221257:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763816.221264:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763816.221279:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763816.221300:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699513089024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763816.221320:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513088896:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763816.221340:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763816.221347:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763816.221350:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763816.221358:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3790 00000100:00080000:2.0:1789763816.221362:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763816.221368:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763816.221376:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763816.221406:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699513089152/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763816.221413:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763816.221417:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763816.221419:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763816.221424:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3790 00000100:00080000:2.0:1789763816.221427:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763816.221430:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763816.221434:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763816.221443:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699513089280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763816.221482:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763816.221589:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763816/real 1789763816] req@ffff8dc00368dc00 x1876699513088896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763816.221635:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699513088896/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763816.221652:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763816.221659:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763816.221664:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513088896:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763816.221715:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513089024:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763816.221784:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763816.221833:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763816/real 1789763816] req@ffff8dc00368c700 x1876699513089024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763816.221872:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699513089024/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763816.221896:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763816.222003:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763816.222014:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513089024:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763816.222032:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513089152:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763816.222082:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763816.222109:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763816/real 1789763816] req@ffff8dc00368e680 x1876699513089152/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763816.222132:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699513089152/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763816.222160:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763816.222167:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763816.222175:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513089152:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763816.222189:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513089280:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763816.222236:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763816.222260:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763816/real 1789763816] req@ffff8dc00368d180 x1876699513089280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763816.222283:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699513089280/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763816.222298:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763816.222305:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763816.222312:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513089280:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763817.245184:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763818.269144:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763819.293082:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763820.317086:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763821.341024:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763821.341036:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763821.341041:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763821.341055:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3795 00000100:00080000:2.0:1789763821.341067:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763821.341075:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763821.341082:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763821.341122:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699513089408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763821.341147:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763821.341150:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763821.341152:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763821.341155:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3795 00000100:00080000:2.0:1789763821.341158:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763821.341161:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763821.341170:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763821.341180:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699513089536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763821.341191:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763821.341196:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763821.341200:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763821.341202:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513089408:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763821.341206:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3795 00000100:00080000:2.0:1789763821.341217:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763821.341222:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763821.341226:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763821.341237:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699513089664/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763821.341245:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763821.341249:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763821.341251:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763821.341255:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3795 00000100:00080000:2.0:1789763821.341260:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763821.341264:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763821.341267:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763821.341276:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699513089792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763821.341287:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763821.341338:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763821/real 1789763821] req@ffff8dc00368df80 x1876699513089408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763821.341351:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699513089408/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763821.341360:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763821.341363:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763821.341367:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513089408:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763821.341386:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513089536:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763821.341402:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763821.341410:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763821/real 1789763821] req@ffff8dc00368f800 x1876699513089536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763821.341418:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699513089536/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763821.341424:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763821.341427:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763821.341429:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513089536:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763821.341433:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513089664:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763821.341444:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763821.341450:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763821/real 1789763821] req@ffff8dc00368e300 x1876699513089664/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763821.341457:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699513089664/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763821.341462:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763821.341464:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763821.341466:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513089664:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763821.341470:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513089792:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763821.341481:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763821.341488:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763821/real 1789763821] req@ffff8dc00368ed80 x1876699513089792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763821.341495:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699513089792/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763821.341516:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763821.341518:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763821.341521:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513089792:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763822.365026:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763823.389049:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763824.413053:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763825.437026:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763826.461010:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763826.461025:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763826.461031:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763826.461046:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3801 00000100:00080000:2.0:1789763826.461059:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763826.461068:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763826.461091:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763826.461119:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699513089920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763826.461151:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763826.461156:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763826.461158:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763826.461163:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3801 00000100:00080000:2.0:1789763826.461167:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763826.461173:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763826.461185:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763826.461196:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513089920:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763826.461205:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699513090048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763826.461220:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763826.461224:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763826.461226:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763826.461230:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3801 00000100:00080000:2.0:1789763826.461235:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763826.461239:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763826.461245:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763826.461259:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699513090176/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763826.461269:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763826.461274:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763826.461277:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1789763826.461282:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789763826.461285:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3801 00000100:00080000:2.0:1789763826.461291:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763826.461297:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763826.461303:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763826.461317:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699513090304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763826.461337:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763826/real 1789763826] req@ffff8dc00368ea00 x1876699513089920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763826.461360:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699513089920/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763826.461372:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763826.461376:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763826.461381:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513089920:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763826.461409:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513090048:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763826.461434:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763826.461447:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763826/real 1789763826] req@ffff8dc00368c000 x1876699513090048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763826.461483:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699513090048/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763826.461493:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763826.461496:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763826.461509:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513090048:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763826.461516:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513090176:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763826.461539:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763826.461552:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763826/real 1789763826] req@ffff8dc00368ce00 x1876699513090176/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763826.461565:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699513090176/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763826.461573:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763826.461577:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763826.461580:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513090176:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763826.461586:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513090304:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763826.461607:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763826.461619:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763826/real 1789763826] req@ffff8dc00368ca80 x1876699513090304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763826.461630:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699513090304/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763826.461645:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763826.461648:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763826.461652:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513090304:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763827.485034:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763828.509099:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763829.533096:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763830.557013:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763831.580997:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763831.581006:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763831.581009:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763831.581017:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3806 00000100:00080000:0.0:1789763831.581026:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763831.581032:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763831.581092:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149f80 x1876699513090432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763831.581131:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763831.581136:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763831.581139:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763831.581144:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3806 00000100:00080000:0.0:1789763831.581150:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763831.581156:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763831.581169:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763831.581192:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149180 x1876699513090560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763831.581194:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513090432:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789763831.581220:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763831.581226:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763831.581228:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763831.581235:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3806 00000100:00080000:0.0:1789763831.581239:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763831.581243:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763831.581250:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763831.581266:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b800 x1876699513090688/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763831.581280:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763831.581285:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763831.581287:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763831.581291:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3806 00000100:00080000:0.0:1789763831.581296:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1789763831.581305:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1789763831.581306:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763831.581319:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763831.581333:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149880 x1876699513090816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1789763831.581397:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763831/real 1789763831] req@ffff8dc005149f80 x1876699513090432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763831.581420:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149f80 x1876699513090432/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:0.0:1789763831.581444:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763831.581445:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763831.581452:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763831.581474:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513090432:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763831.581516:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513090560:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763831.581553:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763831.581572:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763831/real 1789763831] req@ffff8dc005149180 x1876699513090560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763831.581600:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149180 x1876699513090560/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763831.581606:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763831.581608:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763831.581610:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513090560:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763831.581614:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513090688:192.168.203.156@tcp:8:kworker.0 00000800:00000100:2.0:1789763831.581626:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763831.581632:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763831/real 1789763831] req@ffff8dc00514b800 x1876699513090688/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763831.581639:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b800 x1876699513090688/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763831.581675:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763831.581679:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763831.581683:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513090688:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763831.581690:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513090816:192.168.203.156@tcp:38:kworker.0 00000800:00000100:2.0:1789763831.581711:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763831.581723:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763831/real 1789763831] req@ffff8dc005149880 x1876699513090816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763831.581737:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149880 x1876699513090816/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763831.581746:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763831.581750:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763831.581753:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513090816:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763832.605089:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763833.629123:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763834.653122:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763835.677184:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763836.701072:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763836.701083:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763836.701089:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763836.701101:0:12289:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3811 00010000:00010000:1.0:1789763836.701112:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763836.701114:0:12289:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763836.701126:0:12289:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763836.701175:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699513090944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763836.701209:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763836.701213:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763836.701215:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763836.701219:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3811 00000100:00080000:2.0:1789763836.701222:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763836.701225:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763836.701236:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763836.701251:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699513091072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763836.701264:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513090944:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763836.701265:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763836.701292:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763836.701296:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763836.701304:0:12289:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3811 00000100:00080000:2.0:1789763836.701310:0:12289:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763836.701319:0:12289:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763836.701326:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763836.701347:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699513091200/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763836.701366:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789763836.701373:0:12289:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763836.701406:0:12289:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763836.701412:0:12289:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763836.701442:0:12289:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3811 00000100:00100000:0.0:1789763836.701451:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763836/real 1789763836] req@ffff8dc00368fb80 x1876699513090944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763836.701453:0:12289:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763836.701472:0:12289:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763836.701485:0:12289:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763836.701528:0:12289:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699513091328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763836.701534:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699513090944/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763836.701584:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763836.701591:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763836.701597:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513090944:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763836.701627:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513091072:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763836.701654:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763836.701668:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763836/real 1789763836] req@ffff8dc00368f480 x1876699513091072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763836.701680:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699513091072/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763836.701689:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763836.701692:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763836.701695:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513091072:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763836.701702:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513091200:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763836.701739:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763836.701750:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763836/real 1789763836] req@ffff8dc00368f100 x1876699513091200/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763836.701763:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699513091200/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763836.701806:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763836.701809:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763836.701812:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513091200:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763836.701818:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513091328:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763836.701837:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763836.701847:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763836/real 1789763836] req@ffff8dc00368d880 x1876699513091328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763836.701857:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699513091328/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763836.701864:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763836.701874:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763836.701877:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513091328:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763837.725113:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763838.749087:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763839.773209:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763840.797123:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763841.821064:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763841.821082:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763841.821089:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763841.821107:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3816 00000100:00080000:1.0:1789763841.821122:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763841.821133:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763841.821200:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699513091456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:3.0:1789763841.821203:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763841.821244:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763841.821250:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763841.821252:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763841.821257:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3816 00000100:00080000:1.0:1789763841.821262:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763841.821269:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763841.821284:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763841.821302:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699513091584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763841.821316:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513091456:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789763841.821324:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763841.821336:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763841.821339:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763841.821345:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3816 00000100:00080000:1.0:1789763841.821379:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763841.821390:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763841.821399:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763841.821418:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699513091712/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763841.821436:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789763841.821443:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763841.821452:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763841.821458:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763841.821469:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3816 00000100:00080000:1.0:1789763841.821478:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763841.821487:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763841.821499:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763841.821503:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763841/real 1789763841] req@ffff8dc005c94a80 x1876699513091456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763841.821536:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699513091840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763841.821546:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699513091456/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763841.821589:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763841.821597:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763841.821603:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513091456:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763841.821635:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513091584:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763841.821663:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763841.821677:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763841/real 1789763841] req@ffff8dc005c97100 x1876699513091584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763841.821691:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699513091584/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763841.821700:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763841.821704:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763841.821707:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513091584:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763841.821714:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513091712:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763841.821732:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763841.821741:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763841/real 1789763841] req@ffff8dc005c97800 x1876699513091712/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763841.821753:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699513091712/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763841.821762:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763841.821765:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763841.821768:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513091712:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763841.821775:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513091840:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763841.821803:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763841.821821:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763841/real 1789763841] req@ffff8dc005c95c00 x1876699513091840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763841.821843:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699513091840/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763841.821858:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763841.821864:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763841.821870:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513091840:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763842.845255:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763843.869024:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763844.893044:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763845.917221:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763846.941061:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763846.941072:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:0.0:1789763846.941077:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763846.941077:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763846.941093:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3821 00000100:00080000:1.0:1789763846.941105:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763846.941113:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763846.941155:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95180 x1876699513091968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763846.941179:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763846.941182:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763846.941184:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763846.941187:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3821 00000100:00080000:1.0:1789763846.941189:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763846.941192:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763846.941201:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763846.941218:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513091968:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763846.941218:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95c00 x1876699513092096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763846.941242:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763846.941244:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763846.941246:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763846.941252:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3821 00000100:00080000:1.0:1789763846.941255:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763846.941257:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763846.941261:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763846.941272:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97800 x1876699513092224/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763846.941280:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763846.941284:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763846.941285:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763846.941288:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3821 00000100:00080000:1.0:1789763846.941292:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763846.941298:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:0.0:1789763846.941305:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:1.0:1789763846.941324:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763846.941338:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97100 x1876699513092352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763846.941377:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763846/real 1789763846] req@ffff8dc005c95180 x1876699513091968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763846.941399:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95180 x1876699513091968/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763846.941412:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763846.941427:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763846.941430:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513091968:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763846.941453:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513092096:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763846.941473:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763846.941483:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763846/real 1789763846] req@ffff8dc005c95c00 x1876699513092096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763846.941492:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95c00 x1876699513092096/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763846.941498:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763846.941500:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763846.941502:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513092096:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763846.941506:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513092224:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763846.941519:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763846.941526:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763846/real 1789763846] req@ffff8dc005c97800 x1876699513092224/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763846.941534:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97800 x1876699513092224/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763846.941539:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763846.941541:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763846.941543:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513092224:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763846.941547:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513092352:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763846.941557:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763846.941564:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763846/real 1789763846] req@ffff8dc005c97100 x1876699513092352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763846.941571:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97100 x1876699513092352/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763846.941580:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763846.941583:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763846.941585:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513092352:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763847.965194:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763848.989113:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763850.013070:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763851.037131:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763852.061137:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763852.061155:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763852.061168:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763852.061175:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763852.061191:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3826 00000100:00080000:2.0:1789763852.061206:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763852.061217:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763852.061265:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699513092480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763852.061353:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763852.061362:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763852.061366:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763852.061377:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3826 00000100:00080000:2.0:1789763852.061391:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763852.061402:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763852.061417:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763852.061440:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513092480:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763852.061448:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699513092608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763852.061482:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763852.061488:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763852.061491:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763852.061499:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3826 00000100:00080000:2.0:1789763852.061504:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763852.061508:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763852.061515:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763852.061527:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699513092736/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763852.061541:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763852.061546:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763852.061549:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763852.061554:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3826 00000100:00080000:2.0:1789763852.061559:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763852.061563:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763852.061569:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763852.061581:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699513092864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763852.061606:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763852.061696:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763852/real 1789763852] req@ffff8dc00368d880 x1876699513092480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763852.061727:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699513092480/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763852.061744:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763852.061750:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763852.061756:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513092480:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763852.061796:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513092608:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763852.061835:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763852.061854:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763852/real 1789763852] req@ffff8dc00368f100 x1876699513092608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763852.061872:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699513092608/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763852.061886:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763852.061891:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763852.061896:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513092608:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763852.061969:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513092736:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763852.062011:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763852.062053:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763852/real 1789763852] req@ffff8dc00368f480 x1876699513092736/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763852.062088:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699513092736/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763852.062120:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763852.062136:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763852.062151:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513092736:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763852.062170:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513092864:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763852.062219:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763852.062247:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763852/real 1789763852] req@ffff8dc00368fb80 x1876699513092864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763852.062431:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699513092864/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763852.062452:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763852.062461:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763852.062469:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513092864:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763853.085130:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763854.109118:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763855.133128:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763856.157084:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763857.181116:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763857.181130:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763857.181137:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763857.181154:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3831 00000100:00080000:2.0:1789763857.181172:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763857.181184:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763857.181196:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763857.181267:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699513092992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763857.181315:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763857.181322:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763857.181326:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763857.181335:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3831 00000100:00080000:2.0:1789763857.181344:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763857.181353:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763857.181366:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763857.181395:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699513093120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763857.181396:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513092992:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763857.181446:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763857.181450:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763857.181452:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763857.181458:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3831 00000100:00080000:2.0:1789763857.181467:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763857.181472:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763857.181477:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763857.181493:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699513093248/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763857.181510:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763857.181515:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763857.181517:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763857.181526:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3831 00000100:00080000:2.0:1789763857.181530:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763857.181535:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763857.181541:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763857.181550:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763857.181557:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699513093376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763857.181623:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763857/real 1789763857] req@ffff8dc00368ca80 x1876699513092992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763857.181653:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699513092992/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763857.181669:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763857.181675:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763857.181681:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513092992:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763857.181721:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513093120:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763857.181756:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763857.181781:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763857/real 1789763857] req@ffff8dc00368ce00 x1876699513093120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763857.181807:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699513093120/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763857.181827:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763857.181835:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763857.181842:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513093120:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763857.181856:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513093248:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763857.181965:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763857.181983:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763857/real 1789763857] req@ffff8dc00368c000 x1876699513093248/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763857.181999:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699513093248/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763857.182011:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763857.182016:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763857.182020:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513093248:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763857.182028:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513093376:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763857.182051:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763857.182063:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763857/real 1789763857] req@ffff8dc00368ea00 x1876699513093376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763857.182077:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699513093376/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763857.182087:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763857.182091:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763857.182094:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513093376:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763858.205042:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763859.229006:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763860.253181:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763861.277183:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763862.301076:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763862.301093:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763862.301100:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763862.301119:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3836 00000100:00080000:1.0:1789763862.301136:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763862.301149:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789763862.301160:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763862.301233:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94a80 x1876699513093504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763862.301275:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763862.301280:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763862.301283:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763862.301288:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3836 00000100:00080000:1.0:1789763862.301295:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763862.301302:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763862.301315:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763862.301336:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699513093632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763862.301356:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513093504:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789763862.301360:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763862.301381:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763862.301384:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763862.301392:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3836 00000100:00080000:1.0:1789763862.301396:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763862.301402:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763862.301408:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763862.301424:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699513093760/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763862.301438:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763862.301444:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763862.301446:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763862.301452:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3836 00000100:00080000:1.0:1789763862.301468:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763862.301474:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763862.301481:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763862.301490:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763862.301503:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699513093888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763862.301565:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763862/real 1789763862] req@ffff8dc005c94a80 x1876699513093504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763862.301595:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94a80 x1876699513093504/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763862.301609:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763862.301615:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763862.301620:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513093504:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763862.301654:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513093632:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763862.301678:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763862.301691:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763862/real 1789763862] req@ffff8dc005c96a00 x1876699513093632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763862.301703:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699513093632/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763862.301713:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763862.301717:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763862.301720:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513093632:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763862.301726:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513093760:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763862.301744:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763862.301753:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763862/real 1789763862] req@ffff8dc005c94700 x1876699513093760/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763862.301764:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699513093760/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763862.301772:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763862.301775:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763862.301778:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513093760:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763862.301785:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513093888:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763862.301801:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763862.301809:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763862/real 1789763862] req@ffff8dc005c95f80 x1876699513093888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763862.301820:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699513093888/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763862.301828:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763862.301830:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763862.301834:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513093888:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763863.325082:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763864.349083:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763865.373033:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763866.397263:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763867.421066:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763867.421083:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:0.0:1789763867.421087:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763867.421092:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763867.421110:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3842 00000100:00080000:1.0:1789763867.421127:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763867.421139:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763867.421240:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699513094016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763867.421291:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763867.421296:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763867.421299:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763867.421306:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3842 00000100:00080000:1.0:1789763867.421314:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763867.421322:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763867.421339:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763867.421359:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513094016:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763867.421366:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699513094144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763867.421385:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763867.421389:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763867.421391:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763867.421398:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3842 00000100:00080000:1.0:1789763867.421403:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763867.421407:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763867.421414:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763867.421430:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699513094272/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763867.421443:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763867.421448:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763867.421450:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763867.421455:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3842 00000100:00080000:1.0:1789763867.421461:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763867.421466:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763867.421471:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763867.421489:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763867.421489:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699513094400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763867.421579:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763867/real 1789763867] req@ffff8dc005c94380 x1876699513094016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763867.421622:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699513094016/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763867.421645:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763867.421651:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763867.421657:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513094016:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763867.421695:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513094144:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763867.421732:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763867.421750:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763867/real 1789763867] req@ffff8dc005c97b80 x1876699513094144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763867.421767:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699513094144/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763867.421779:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763867.421783:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763867.421788:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513094144:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763867.421795:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513094272:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763867.421842:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763867.421857:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763867/real 1789763867] req@ffff8dc005c96300 x1876699513094272/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763867.421872:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699513094272/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763867.421883:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763867.421887:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763867.421891:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513094272:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763867.421928:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513094400:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763867.421953:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763867.421968:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763867/real 1789763867] req@ffff8dc005c95500 x1876699513094400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763867.421983:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699513094400/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763867.421993:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763867.421997:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763867.422001:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513094400:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763868.445173:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763869.469172:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763870.493088:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763871.517049:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763872.541072:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763872.541092:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763872.541102:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:3.0:1789763872.541118:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763872.541152:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3847 00000100:00080000:2.0:1789763872.541172:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763872.541186:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763872.541260:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699513094528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763872.541303:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763872.541309:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763872.541312:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763872.541318:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3847 00000100:00080000:2.0:1789763872.541325:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763872.541332:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763872.541348:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763872.541380:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513094528:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763872.541386:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699513094656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763872.541395:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763872.541397:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763872.541399:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763872.541402:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3847 00000100:00080000:2.0:1789763872.541405:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763872.541407:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763872.541411:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763872.541418:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699513094784/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763872.541426:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763872.541429:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763872.541430:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763872.541433:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3847 00000100:00080000:2.0:1789763872.541435:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763872.541437:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763872.541441:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763872.541448:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699513094912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763872.541467:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763872.541515:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763872/real 1789763872] req@ffff8dc00368ed80 x1876699513094528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763872.541536:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699513094528/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763872.541546:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763872.541549:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763872.541552:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513094528:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763872.541576:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513094656:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763872.541592:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763872.541606:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763872/real 1789763872] req@ffff8dc00368e300 x1876699513094656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763872.541615:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699513094656/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763872.541622:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763872.541624:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763872.541626:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513094656:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763872.541631:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513094784:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763872.541643:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763872.541650:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763872/real 1789763872] req@ffff8dc00368f800 x1876699513094784/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763872.541658:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699513094784/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763872.541668:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763872.541671:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763872.541673:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513094784:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763872.541693:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513094912:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763872.541727:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763872.541748:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763872/real 1789763872] req@ffff8dc00368df80 x1876699513094912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763872.541774:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699513094912/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763872.541793:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763872.541800:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763872.541807:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513094912:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763873.565070:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763874.589070:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763875.613174:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763876.637039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763877.661080:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763877.661131:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763877.661139:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763877.661162:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3852 00000100:00080000:2.0:1789763877.661178:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763877.661191:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763877.661238:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763877.661272:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699513095040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763877.661328:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763877.661333:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763877.661336:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763877.661342:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3852 00000100:00080000:2.0:1789763877.661348:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763877.661356:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763877.661370:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763877.661390:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513095040:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763877.661395:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699513095168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763877.661417:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763877.661421:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763877.661424:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763877.661429:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3852 00000100:00080000:2.0:1789763877.661434:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763877.661439:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763877.661445:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763877.661462:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699513095296/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763877.661484:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763877.661490:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763877.661493:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763877.661497:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3852 00000100:00080000:2.0:1789763877.661502:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763877.661507:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763877.661513:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763877.661529:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699513095424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763877.661581:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513095168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763877.661667:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513095296:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763877.661746:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513095424:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763878.685236:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763879.709064:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763880.733304:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1789763880.733457:0:2348:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:0.0:1789763880.733534:0:2348:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1789763880.733560:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789763880.733637:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789763880.733654:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:0.0:1789763880.733677:0:2348:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:2.0:1789763880.733753:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763877/real 1789763880] req@ffff8dc00368e680 x1876699513095168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763880.733855:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699513095168/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763880.733876:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763880.733884:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763880.733891:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513095168:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763880.733906:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763877/real 1789763880] req@ffff8dc00368c700 x1876699513095296/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763880.733993:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699513095296/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763880.734008:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763880.734017:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763880.734024:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513095296:192.168.203.156@tcp:8:kworker.0 00000100:00100000:2.0:1789763880.734039:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763877/real 1789763880] req@ffff8dc00368dc00 x1876699513095424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763880.734083:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699513095424/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763880.734097:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763880.734104:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763880.734110:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513095424:192.168.203.156@tcp:38:kworker.0 00000100:00100000:2.0:1789763880.734125:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763877/real 1789763880] req@ffff8dc00368d180 x1876699513095040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763880.734146:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699513095040/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763880.734158:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1789763880.734163:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:2.0:1789763880.734169:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513095040:192.168.203.156@tcp:250:kworker.0 00010000:00010000:1.0:1789763881.757046:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763882.781105:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763882.781123:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763882.781137:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763882.781144:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763882.781159:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3857 00000100:00080000:2.0:1789763882.781175:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763882.781186:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763882.781254:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699513095552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763882.781322:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763882.781331:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763882.781335:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763882.781344:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3857 00000100:00080000:2.0:1789763882.781368:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763882.781375:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763882.781403:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763882.781426:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699513095680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763882.781442:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763882.781448:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763882.781451:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763882.781456:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3857 00000100:00080000:2.0:1789763882.781463:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763882.781469:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763882.781478:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763882.781496:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699513095808/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763882.781565:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763882.781571:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763882.781578:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763882.781580:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513095552:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763882.781592:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3857 00000100:00080000:2.0:1789763882.781602:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763882.781609:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763882.781628:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763882.781654:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699513095936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763882.781730:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763882.781816:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763882/real 1789763882] req@ffff8dc00368d180 x1876699513095552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763882.781848:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699513095552/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763882.781867:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763882.781874:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763882.781882:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513095552:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763882.781896:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513095680:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763882.781976:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763882.782000:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763882/real 1789763882] req@ffff8dc00368dc00 x1876699513095680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763882.782026:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699513095680/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763882.782041:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763882.782073:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763882.782080:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513095680:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763882.782218:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513095808:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763882.782262:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763882.782282:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763882/real 1789763882] req@ffff8dc00368c700 x1876699513095808/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763882.782302:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699513095808/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763882.782315:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763882.782321:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763882.782326:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513095808:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763882.782336:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513095936:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763882.782370:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763882.782396:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763882/real 1789763882] req@ffff8dc00368e680 x1876699513095936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763882.782415:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699513095936/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763882.782428:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763882.782433:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763882.782438:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513095936:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763883.805232:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763884.829200:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763885.853161:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763886.877183:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763887.901207:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763887.901222:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763887.901229:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1789763887.901248:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763887.901249:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3862 00000100:00080000:0.0:1789763887.901272:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763887.901283:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763887.901348:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149c00 x1876699513096064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763887.901391:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763887.901398:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763887.901401:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763887.901412:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3862 00000100:00080000:0.0:1789763887.901418:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763887.901429:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763887.901447:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763887.901463:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513096064:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763887.901485:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148380 x1876699513096192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763887.901506:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763887.901512:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763887.901516:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763887.901523:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3862 00000100:00080000:0.0:1789763887.901530:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763887.901547:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763887.901561:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763887.901589:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148000 x1876699513096320/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763887.901615:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1789763887.901643:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763887.901653:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763887.901658:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763887.901671:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3862 00000100:00080000:0.0:1789763887.901681:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763887.901689:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763887.901698:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763887/real 1789763887] req@ffff8dc005149c00 x1876699513096064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789763887.901701:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763887.901731:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b480 x1876699513096448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763887.901739:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149c00 x1876699513096064/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763887.901775:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763887.901784:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763887.901790:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513096064:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763887.901846:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513096192:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763887.901888:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763887.901968:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763887/real 1789763887] req@ffff8dc005148380 x1876699513096192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763887.901990:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148380 x1876699513096192/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763887.902002:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763887.902007:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763887.902038:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513096192:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763887.902051:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513096320:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763887.902088:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763887.902103:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763887/real 1789763887] req@ffff8dc005148000 x1876699513096320/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763887.902120:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148000 x1876699513096320/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763887.902131:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763887.902135:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763887.902139:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513096320:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763887.902146:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513096448:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763887.902173:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763887.902186:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763887/real 1789763887] req@ffff8dc00514b480 x1876699513096448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763887.902204:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b480 x1876699513096448/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763887.902224:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763887.902228:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763887.902234:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513096448:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763888.925184:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763889.949061:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763890.973179:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763891.997224:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763893.021086:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763893.021095:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763893.021098:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763893.021108:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3867 00000100:00080000:1.0:1789763893.021117:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763893.021122:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789763893.021138:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763893.021157:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699513096576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763893.021181:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763893.021184:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763893.021185:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763893.021188:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3867 00000100:00080000:1.0:1789763893.021190:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763893.021193:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763893.021200:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763893.021209:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699513096704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763893.021232:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513096576:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789763893.021259:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763893.021264:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763893.021266:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763893.021273:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3867 00000100:00080000:1.0:1789763893.021279:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763893.021284:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763893.021291:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763893.021307:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699513096832/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763893.021321:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763893.021329:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1789763893.021330:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789763893.021333:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763893.021342:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3867 00000100:00080000:1.0:1789763893.021349:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763893.021355:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763893.021362:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763893.021377:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699513096960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763893.021413:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763893/real 1789763893] req@ffff8dc005c95880 x1876699513096576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763893.021442:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699513096576/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763893.021463:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763893.021471:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763893.021479:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513096576:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763893.021530:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513096704:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763893.021574:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763893.021593:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763893/real 1789763893] req@ffff8dc005c94e00 x1876699513096704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763893.021610:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699513096704/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763893.021621:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763893.021625:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763893.021629:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513096704:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763893.021637:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513096832:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763893.021659:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763893.021671:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763893/real 1789763893] req@ffff8dc005c97480 x1876699513096832/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763893.021685:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699513096832/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763893.021695:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763893.021698:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763893.021702:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513096832:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763893.021709:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513096960:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763893.021740:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763893.021751:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763893/real 1789763893] req@ffff8dc005c96680 x1876699513096960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763893.021765:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699513096960/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763893.021794:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763893.021798:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763893.021802:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513096960:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763894.045178:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763895.069270:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763896.093177:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763897.117189:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763898.141181:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763898.141193:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763898.141199:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763898.141214:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3872 00010000:00010000:1.0:1789763898.141223:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763898.141228:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763898.141240:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763898.141293:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e680 x1876699513097088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763898.141328:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763898.141344:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763898.141346:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763898.141352:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3872 00000100:00080000:2.0:1789763898.141358:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763898.141365:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763898.141378:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763898.141396:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513097088:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763898.141400:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c700 x1876699513097216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763898.141423:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763898.141427:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763898.141430:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763898.141436:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3872 00000100:00080000:2.0:1789763898.141440:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763898.141446:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763898.141456:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763898.141480:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368dc00 x1876699513097344/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763898.141493:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:2.0:1789763898.141532:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763898.141541:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763898.141546:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1789763898.141550:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763898/real 1789763898] req@ffff8dc00368e680 x1876699513097088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763898.141561:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3872 00000100:00080000:2.0:1789763898.141579:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763898.141589:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763898.141595:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e680 x1876699513097088/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1789763898.141605:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1789763898.141614:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763898.141622:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763898.141627:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513097088:192.168.203.156@tcp:250:kworker.0 00000100:00100000:2.0:1789763898.141658:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d180 x1876699513097472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763898.141675:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513097216:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763898.141721:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763898.141744:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763898/real 1789763898] req@ffff8dc00368c700 x1876699513097216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763898.141761:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c700 x1876699513097216/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763898.141771:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763898.141775:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763898.141780:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513097216:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763898.141788:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513097344:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763898.141809:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763898.141820:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763898/real 1789763898] req@ffff8dc00368dc00 x1876699513097344/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763898.141834:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368dc00 x1876699513097344/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763898.141844:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763898.141847:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763898.141851:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513097344:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763898.141867:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513097472:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763898.141970:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763898.142009:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763898/real 1789763898] req@ffff8dc00368d180 x1876699513097472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763898.142035:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d180 x1876699513097472/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763898.142078:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763898.142092:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763898.142098:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513097472:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763899.165122:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763900.189227:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763901.213195:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763902.237082:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763903.261049:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763903.261070:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763903.261078:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763903.261099:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3877 00000100:00080000:3.0:1789763903.261117:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763903.261132:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789763903.261157:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763903.261202:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aca80 x1876699513097600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763903.261248:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763903.261256:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763903.261259:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763903.261269:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3877 00000100:00080000:3.0:1789763903.261276:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763903.261285:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763903.261298:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513097600:192.168.203.156@tcp:250:kworker.0 00000008:00100000:3.0:1789763903.261302:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763903.261331:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adc00 x1876699513097728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763903.261352:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763903.261358:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763903.261363:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763903.261370:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3877 00000100:00080000:3.0:1789763903.261379:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763903.261387:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763903.261396:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789763903.261405:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763903.261426:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699513097856/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763903.261447:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763903.261453:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763903.261456:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763903.261465:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3877 00000100:00080000:3.0:1789763903.261471:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1789763903.261474:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763903/real 1789763903] req@ffff8dc0107aca80 x1876699513097600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763903.261484:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763903.261502:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763903.261512:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aca80 x1876699513097600/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763903.261531:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1789763903.261531:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699513097984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763903.261542:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763903.261556:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513097600:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763903.261595:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513097728:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763903.261621:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763903.261636:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763903/real 1789763903] req@ffff8dc0107adc00 x1876699513097728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763903.261649:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adc00 x1876699513097728/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763903.261659:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763903.261662:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763903.261666:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513097728:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763903.261671:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513097856:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763903.261689:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763903.261698:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763903/real 1789763903] req@ffff8dc0107ae300 x1876699513097856/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763903.261711:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699513097856/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763903.261719:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763903.261722:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763903.261725:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513097856:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763903.261730:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513097984:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763903.261746:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763903.261755:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763903/real 1789763903] req@ffff8dc0107afb80 x1876699513097984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763903.261767:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699513097984/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763903.261775:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763903.261778:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763903.261781:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513097984:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763904.285188:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763905.309119:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763906.333104:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763907.357193:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763908.381053:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763908.381066:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763908.381074:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763908.381089:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3882 00000100:00080000:3.0:1789763908.381102:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763908.381113:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:0.0:1789763908.381116:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763908.381182:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699513098112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763908.381230:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763908.381235:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763908.381238:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763908.381243:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3882 00000100:00080000:3.0:1789763908.381248:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763908.381253:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763908.381266:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763908.381283:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699513098240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763908.381301:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763908.381309:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1789763908.381312:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513098112:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763908.381315:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763908.381339:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3882 00000100:00080000:3.0:1789763908.381344:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763908.381349:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763908.381356:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763908.381375:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699513098368/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763908.381389:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763908.381394:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763908.381396:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763908.381401:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3882 00000100:00080000:3.0:1789763908.381409:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763908.381415:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763908.381421:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763908.381434:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699513098496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763908.381457:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763908.381543:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763908/real 1789763908] req@ffff8dc0107ac000 x1876699513098112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763908.381586:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699513098112/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763908.381603:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763908.381609:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763908.381616:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513098112:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763908.381653:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513098240:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763908.381680:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763908.381696:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763908/real 1789763908] req@ffff8dc0107ad500 x1876699513098240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763908.381712:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699513098240/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763908.381723:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763908.381727:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763908.381731:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513098240:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763908.381739:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513098368:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763908.381760:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763908.381772:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763908/real 1789763908] req@ffff8dc0107af480 x1876699513098368/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763908.381786:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699513098368/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763908.381805:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763908.381809:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763908.381813:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513098368:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763908.381820:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513098496:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763908.381840:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763908.381851:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763908/real 1789763908] req@ffff8dc0107ad180 x1876699513098496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763908.381948:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699513098496/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763908.381961:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763908.381965:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763908.381970:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513098496:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763909.405165:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763910.429137:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763911.453079:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763912.477072:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763913.501038:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763913.501051:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763913.501057:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763913.501071:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3888 00000100:00080000:0.0:1789763913.501085:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763913.501094:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763913.501142:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514bb80 x1876699513098624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763913.501185:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763913.501190:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763913.501193:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763913.501198:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3888 00000100:00080000:0.0:1789763913.501202:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763913.501207:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763913.501218:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763913.501232:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514ad80 x1876699513098752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763913.501246:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763913.501251:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763913.501253:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763913.501259:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3888 00000100:00080000:0.0:1789763913.501263:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763913.501268:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763913.501273:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763913.501285:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148700 x1876699513098880/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763913.501302:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513098624:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789763913.501304:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763913.501319:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763913.501327:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763913.501334:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3888 00000100:00080000:0.0:1789763913.501338:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763913.501343:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763913.501348:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763913.501365:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514aa00 x1876699513099008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763913.501424:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763913.501487:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763913/real 1789763913] req@ffff8dc00514bb80 x1876699513098624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00010000:00010000:2.0:1789763913.501497:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763913.501515:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514bb80 x1876699513098624/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763913.501535:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763913.501541:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763913.501546:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513098624:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763913.501557:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513098752:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763913.501582:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763913.501598:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763913/real 1789763913] req@ffff8dc00514ad80 x1876699513098752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763913.501612:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514ad80 x1876699513098752/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763913.501622:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763913.501626:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763913.501630:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513098752:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763913.501661:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513098880:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763913.501685:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763913.501697:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763913/real 1789763913] req@ffff8dc005148700 x1876699513098880/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763913.501711:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148700 x1876699513098880/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763913.501721:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763913.501725:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763913.501728:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513098880:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763913.501735:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513099008:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763913.501754:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763913.501766:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763913/real 1789763913] req@ffff8dc00514aa00 x1876699513099008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763913.501780:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514aa00 x1876699513099008/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763913.501789:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763913.501793:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763913.501796:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513099008:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763914.525050:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763915.549155:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763916.573159:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763917.597087:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763918.621030:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763918.621039:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763918.621043:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789763918.621044:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763918.621054:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3893 00000100:00080000:1.0:1789763918.621066:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763918.621076:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763918.621111:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96d80 x1876699513099136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763918.621137:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763918.621141:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763918.621142:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763918.621145:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3893 00000100:00080000:1.0:1789763918.621148:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763918.621151:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763918.621159:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763918.621175:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513099136:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763918.621175:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96680 x1876699513099264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763918.621191:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763918.621193:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763918.621195:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763918.621198:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3893 00000100:00080000:1.0:1789763918.621201:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763918.621203:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763918.621207:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763918.621217:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97480 x1876699513099392/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763918.621224:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763918.621227:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763918.621229:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763918.621233:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3893 00000100:00080000:1.0:1789763918.621236:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763918.621238:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763918.621242:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763918.621245:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763918.621254:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94e00 x1876699513099520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763918.621311:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763918/real 1789763918] req@ffff8dc005c96d80 x1876699513099136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763918.621334:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96d80 x1876699513099136/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763918.621346:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763918.621365:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763918.621368:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513099136:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763918.621390:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513099264:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763918.621410:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763918.621437:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763918/real 1789763918] req@ffff8dc005c96680 x1876699513099264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763918.621446:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96680 x1876699513099264/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763918.621453:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763918.621455:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763918.621457:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513099264:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763918.621461:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513099392:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763918.621474:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763918.621496:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763918/real 1789763918] req@ffff8dc005c97480 x1876699513099392/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763918.621505:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97480 x1876699513099392/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763918.621511:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763918.621514:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763918.621517:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513099392:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763918.621524:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513099520:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763918.621563:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763918.621575:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763918/real 1789763918] req@ffff8dc005c94e00 x1876699513099520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763918.621588:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94e00 x1876699513099520/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763918.621594:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763918.621596:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763918.621599:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513099520:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763919.645116:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763920.669054:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763921.693070:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763922.717122:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763923.741031:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763923.741045:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1789763923.741051:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789763923.741059:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763923.741067:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3898 00000100:00080000:1.0:1789763923.741082:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763923.741092:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763923.741146:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95880 x1876699513099648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763923.741182:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763923.741186:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763923.741188:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763923.741193:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3898 00000100:00080000:1.0:1789763923.741197:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763923.741202:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763923.741215:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763923.741231:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513099648:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763923.741233:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95500 x1876699513099776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763923.741260:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763923.741263:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763923.741264:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763923.741269:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3898 00000100:00080000:1.0:1789763923.741273:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763923.741277:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763923.741282:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763923.741299:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96300 x1876699513099904/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763923.741316:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763923.741343:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763923.741346:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763923.741352:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3898 00000100:00080000:1.0:1789763923.741357:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763923.741363:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763923.741370:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1789763923.741372:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763923.741407:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c97b80 x1876699513100032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763923.741426:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763923/real 1789763923] req@ffff8dc005c95880 x1876699513099648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763923.741449:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95880 x1876699513099648/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763923.741462:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763923.741466:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763923.741473:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513099648:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763923.741504:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513099776:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763923.741532:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763923.741545:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763923/real 1789763923] req@ffff8dc005c95500 x1876699513099776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763923.741556:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95500 x1876699513099776/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763923.741566:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763923.741570:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763923.741574:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513099776:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763923.741584:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513099904:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763923.741606:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763923.741618:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763923/real 1789763923] req@ffff8dc005c96300 x1876699513099904/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763923.741630:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96300 x1876699513099904/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763923.741638:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763923.741643:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763923.741646:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513099904:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763923.741652:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513100032:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763923.741683:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763923.741702:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763923/real 1789763923] req@ffff8dc005c97b80 x1876699513100032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763923.741720:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c97b80 x1876699513100032/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763923.741733:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763923.741738:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763923.741743:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513100032:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763924.765091:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763925.789231:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763926.813024:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763927.837125:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763928.861135:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763928.861150:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00010000:00010000:2.0:1789763928.861151:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1789763928.861159:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763928.861176:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3903 00000100:00080000:1.0:1789763928.861192:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763928.861202:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763928.861261:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94380 x1876699513100160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763928.861337:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763928.861345:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763928.861348:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763928.861355:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3903 00000100:00080000:1.0:1789763928.861361:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763928.861367:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763928.861379:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763928.861415:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c95f80 x1876699513100288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763928.861431:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513100160:192.168.203.156@tcp:250:kworker.0 00000100:00080000:1.0:1789763928.861441:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763928.861455:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1789763928.861457:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763928.861464:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3903 00000100:00080000:1.0:1789763928.861473:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763928.861485:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763928.861514:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763928.861537:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c94700 x1876699513100416/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763928.861558:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1789763928.861566:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1789763928.861570:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1789763928.861581:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3903 00000800:00000100:0.0:1789763928.861588:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:1.0:1789763928.861593:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1789763928.861626:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:1.0:1789763928.861636:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763928.861657:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005c96a00 x1876699513100544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763928.861751:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763928/real 1789763928] req@ffff8dc005c94380 x1876699513100160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763928.861792:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94380 x1876699513100160/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763928.861810:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763928.861816:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763928.861822:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513100160:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763928.861862:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513100288:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763928.861951:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763928.861981:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763928/real 1789763928] req@ffff8dc005c95f80 x1876699513100288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763928.862007:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c95f80 x1876699513100288/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763928.862026:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763928.862033:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763928.862040:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513100288:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763928.862055:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513100416:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763928.862089:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763928.862106:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763928/real 1789763928] req@ffff8dc005c94700 x1876699513100416/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763928.862131:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c94700 x1876699513100416/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763928.862142:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763928.862146:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763928.862150:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513100416:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763928.862158:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513100544:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763928.862180:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763928.862212:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763928/real 1789763928] req@ffff8dc005c96a00 x1876699513100544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763928.862227:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005c96a00 x1876699513100544/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763928.862237:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763928.862242:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763928.862246:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005c96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513100544:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763929.885085:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763930.909116:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763931.933182:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763932.957107:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763933.981097:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763933.981109:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763933.981114:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763933.981129:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3908 00000100:00080000:0.0:1789763933.981142:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763933.981152:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763933.981225:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148a80 x1876699513100672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763933.981269:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763933.981276:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763933.981280:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763933.981290:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3908 00000100:00080000:0.0:1789763933.981298:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763933.981331:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763933.981339:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513100672:192.168.203.156@tcp:250:kworker.0 00000008:00100000:0.0:1789763933.981354:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763933.981398:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514aa00 x1876699513100800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763933.981416:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763933.981421:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763933.981425:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763933.981431:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3908 00000100:00080000:0.0:1789763933.981437:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1789763933.981445:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:0.0:1789763933.981450:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763933.981462:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763933.981489:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148700 x1876699513100928/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763933.981504:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763933.981508:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763933.981511:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763933.981516:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3908 00000100:00080000:0.0:1789763933.981521:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763933.981527:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:1.0:1789763933.981535:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763933/real 1789763933] req@ffff8dc005148a80 x1876699513100672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1789763933.981539:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763933.981567:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514ad80 x1876699513101056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763933.981579:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148a80 x1876699513100672/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763933.981610:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763933.981616:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763933.981623:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513100672:192.168.203.156@tcp:250:kworker.0 00010000:00010000:0.0:1789763933.981651:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763933.981659:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513100800:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763933.981734:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763933.981753:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763933/real 1789763933] req@ffff8dc00514aa00 x1876699513100800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763933.981769:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514aa00 x1876699513100800/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763933.981791:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763933.981795:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763933.981800:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513100800:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763933.981807:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513100928:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763933.981828:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763933.981840:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763933/real 1789763933] req@ffff8dc005148700 x1876699513100928/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763933.981864:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148700 x1876699513100928/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763933.981874:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763933.981877:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763933.981881:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513100928:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763933.981888:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513101056:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763933.981953:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763933.981968:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763933/real 1789763933] req@ffff8dc00514ad80 x1876699513101056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763933.981983:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514ad80 x1876699513101056/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763933.981993:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763933.981997:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763933.982002:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513101056:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763935.005176:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763936.029132:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763937.053167:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763938.077234:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763939.101076:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763939.101098:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763939.101107:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763939.101129:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3913 00010000:00010000:0.0:1789763939.101138:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763939.101150:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763939.101168:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763939.101243:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699513101184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763939.101305:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763939.101314:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763939.101319:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763939.101330:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3913 00000100:00080000:2.0:1789763939.101340:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763939.101350:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763939.101369:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513101184:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789763939.101370:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763939.101408:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699513101312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763939.101433:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763939.101440:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763939.101444:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763939.101453:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3913 00000100:00080000:2.0:1789763939.101460:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763939.101469:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763939.101481:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1789763939.101491:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789763939.101513:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699513101440/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763939.101537:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763939.101544:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763939.101549:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763939.101561:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3913 00000100:00100000:0.0:1789763939.101562:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763939/real 1789763939] req@ffff8dc00368df80 x1876699513101184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789763939.101572:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763939.101588:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763939.101603:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763939.101606:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699513101184/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763939.101629:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1789763939.101660:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699513101568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763939.101670:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763939.101690:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513101184:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763939.101768:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513101312:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763939.101821:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763939.101849:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763939/real 1789763939] req@ffff8dc00368f800 x1876699513101312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763939.101891:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699513101312/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763939.101940:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763939.101945:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763939.101950:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513101312:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763939.101959:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513101440:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763939.101995:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763939.102011:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763939/real 1789763939] req@ffff8dc00368e300 x1876699513101440/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763939.102027:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699513101440/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763939.102038:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763939.102042:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763939.102046:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513101440:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763939.102053:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513101568:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763939.102074:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763939.102085:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763939/real 1789763939] req@ffff8dc00368ed80 x1876699513101568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763939.102100:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699513101568/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763939.102118:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763939.102122:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763939.102126:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513101568:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763940.125084:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763941.149230:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763942.173075:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763943.197128:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763944.221038:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763944.221051:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763944.221056:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763944.221070:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3918 00000100:00080000:3.0:1789763944.221083:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763944.221093:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763944.221141:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699513101696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763944.221179:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763944.221185:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763944.221189:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1789763944.221194:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763944.221199:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3918 00000100:00080000:3.0:1789763944.221208:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763944.221216:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763944.221229:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763944.221242:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513101696:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763944.221252:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699513101824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763944.221270:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763944.221276:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763944.221278:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763944.221283:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3918 00000100:00080000:3.0:1789763944.221287:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763944.221293:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763944.221301:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763944.221318:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699513101952/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763944.221332:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763944.221336:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763944.221338:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763944.221343:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3918 00000100:00080000:3.0:1789763944.221348:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763944.221366:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763944.221374:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763944.221393:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699513102080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1789763944.221399:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513101824:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763944.221460:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513101952:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763944.221493:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513102080:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763945.245066:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763946.269077:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763947.293143:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1789763947.293344:0:2345:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.156:988 00000400:00000100:2.0:1789763947.293408:0:2345:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.156@tcp at host 192.168.203.156 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1789763947.293434:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789763947.293496:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789763947.293514:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000800:00000100:2.0:1789763947.293533:0:2345:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.156@tcp 00000100:00100000:0.0:1789763947.293621:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763944/real 1789763947] req@ffff8dc0107ac380 x1876699513102080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763947.293675:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699513102080/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763947.293700:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763947.293710:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763947.293719:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513102080:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763947.293737:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763944/real 1789763947] req@ffff8dc0107aed80 x1876699513101824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763947.293802:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699513101824/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763947.293820:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763947.293828:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763947.293835:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513101824:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763947.293848:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763944/real 1789763947] req@ffff8dc0107adf80 x1876699513101952/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763947.293873:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699513101952/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763947.293890:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763947.293898:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763947.293956:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513101952:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763947.293974:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763944/real 1789763947] req@ffff8dc0107ac700 x1876699513101696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763947.294004:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699513101696/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763947.294021:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763947.294028:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763947.294036:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513101696:192.168.203.156@tcp:250:kworker.0 00010000:00010000:0.0:1789763948.317075:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763949.341087:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763949.341173:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763949.341182:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763949.341186:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763949.341199:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3923 00000100:00080000:2.0:1789763949.341221:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763949.341229:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763949.341277:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699513102208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763949.341320:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763949.341325:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763949.341328:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763949.341333:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3923 00000100:00080000:2.0:1789763949.341338:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763949.341343:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763949.341359:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763949.341378:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699513102336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763949.341413:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1789763949.341414:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513102208:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763949.341425:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763949.341430:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763949.341437:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3923 00000100:00080000:2.0:1789763949.341444:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763949.341450:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763949.341474:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763949.341490:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699513102464/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763949.341503:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763949.341507:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763949.341510:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763949.341515:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3923 00000100:00080000:2.0:1789763949.341532:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763949.341537:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763949.341543:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763949.341582:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699513102592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1789763949.341589:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763949.341658:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763949/real 1789763949] req@ffff8dc00368ea00 x1876699513102208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763949.341686:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699513102208/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763949.341701:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763949.341706:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763949.341711:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513102208:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763949.341755:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513102336:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763949.341785:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763949.341802:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763949/real 1789763949] req@ffff8dc00368c000 x1876699513102336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763949.341816:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699513102336/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763949.341826:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763949.341830:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763949.341834:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513102336:192.168.203.156@tcp:38:kworker.0 00000100:00100000:0.0:1789763949.341841:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513102464:192.168.203.156@tcp:8:kworker.0 00000800:00000100:0.0:1789763949.341866:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763949.341879:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763949/real 1789763949] req@ffff8dc00368ce00 x1876699513102464/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763949.341911:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699513102464/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763949.341985:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763949.341989:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763949.341994:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513102464:192.168.203.156@tcp:8:kworker.0 00000100:00100000:0.0:1789763949.342002:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513102592:192.168.203.156@tcp:38:kworker.0 00000800:00000100:0.0:1789763949.342029:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:0.0:1789763949.342040:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763949/real 1789763949] req@ffff8dc00368ca80 x1876699513102592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1789763949.342051:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699513102592/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1789763949.342057:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1789763949.342060:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:0.0:1789763949.342063:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513102592:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763950.365097:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763951.389061:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763952.413047:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763953.437082:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763954.461091:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763954.461105:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763954.461110:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763954.461124:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3929 00000100:00080000:0.0:1789763954.461138:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763954.461147:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763954.461202:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514bb80 x1876699513102720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763954.461240:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763954.461245:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763954.461246:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763954.461251:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3929 00000100:00080000:0.0:1789763954.461254:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763954.461257:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763954.461269:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763954.461282:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b480 x1876699513102848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763954.461291:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763954.461295:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763954.461298:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763954.461301:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3929 00000100:00080000:0.0:1789763954.461304:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763954.461325:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763954.461332:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1789763954.461345:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148000 x1876699513102976/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763954.461354:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513102720:192.168.203.156@tcp:250:kworker.0 00000100:00080000:0.0:1789763954.461360:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763954.461376:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763954.461378:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763954.461383:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3929 00000100:00080000:0.0:1789763954.461386:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763954.461391:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763954.461395:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763954.461429:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005148380 x1876699513103104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763954.461489:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00010000:00010000:0.0:1789763954.461514:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763954.461592:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763954/real 1789763954] req@ffff8dc00514bb80 x1876699513102720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763954.461633:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514bb80 x1876699513102720/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763954.461651:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763954.461658:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763954.461665:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513102720:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763954.461678:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513102848:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763954.461709:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763954.461727:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763954/real 1789763954] req@ffff8dc00514b480 x1876699513102848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763954.461746:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b480 x1876699513102848/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763954.461760:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763954.461766:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763954.461771:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513102848:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763954.461814:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513102976:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763954.461841:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763954.461856:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763954/real 1789763954] req@ffff8dc005148000 x1876699513102976/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763954.461875:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148000 x1876699513102976/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763954.461888:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763954.461893:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763954.461898:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513102976:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763954.461965:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513103104:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763954.461996:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763954.462014:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763954/real 1789763954] req@ffff8dc005148380 x1876699513103104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763954.462040:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005148380 x1876699513103104/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763954.462052:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763954.462056:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763954.462061:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005148380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513103104:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763955.485075:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763956.509192:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763957.533188:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763958.557145:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1789763959.581068:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763959.581082:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1789763959.581088:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763959.581101:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3934 00000100:00080000:0.0:1789763959.581115:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763959.581125:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:0.0:1789763959.581173:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149c00 x1876699513103232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763959.581224:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763959.581229:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763959.581232:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763959.581237:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3934 00000100:00080000:0.0:1789763959.581241:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763959.581245:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763959.581257:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763959.581274:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149880 x1876699513103360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763959.581289:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763959.581293:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1789763959.581296:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763959.581302:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3934 00000100:00080000:0.0:1789763959.581305:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763959.581309:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763959.581315:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1789763959.581330:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513103232:192.168.203.156@tcp:250:kworker.0 00000100:00100000:0.0:1789763959.581334:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00514b800 x1876699513103488/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1789763959.581364:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1789763959.581368:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1789763959.581371:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1789763959.581376:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3934 00000100:00080000:0.0:1789763959.581379:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1789763959.581386:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:0.0:1789763959.581392:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1789763959.581404:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc005149180 x1876699513103616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763959.581463:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763959.581571:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763959/real 1789763959] req@ffff8dc005149c00 x1876699513103232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763959.581603:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149c00 x1876699513103232/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763959.581624:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763959.581633:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763959.581638:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513103232:192.168.203.156@tcp:250:kworker.0 00010000:00010000:3.0:1789763959.581653:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1789763959.581657:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513103360:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763959.581716:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763959.581745:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763959/real 1789763959] req@ffff8dc005149880 x1876699513103360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763959.581779:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149880 x1876699513103360/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763959.581790:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763959.581795:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763959.581799:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513103360:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763959.581839:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513103488:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763959.581879:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763959.581981:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763959/real 1789763959] req@ffff8dc00514b800 x1876699513103488/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763959.582010:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00514b800 x1876699513103488/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763959.582025:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763959.582049:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763959.582059:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00514b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513103488:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763959.582077:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513103616:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763959.582128:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763959.582148:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763959/real 1789763959] req@ffff8dc005149180 x1876699513103616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763959.582183:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc005149180 x1876699513103616/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763959.582192:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763959.582195:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763959.582198:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc005149180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513103616:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763960.605055:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763961.629075:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763962.653189:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763963.677067:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763964.701147:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763964.701163:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763964.701170:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:0.0:1789763964.701186:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763964.701189:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3939 00000100:00080000:2.0:1789763964.701209:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763964.701220:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:2.0:1789763964.701287:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699513103744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763964.701331:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763964.701337:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763964.701340:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763964.701345:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3939 00000100:00080000:2.0:1789763964.701350:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763964.701354:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763964.701368:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763964.701384:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699513103872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763964.701397:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763964.701403:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763964.701405:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763964.701410:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3939 00000100:00080000:2.0:1789763964.701414:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763964.701418:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763964.701423:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763964.701435:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699513104000/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763964.701476:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763964.701483:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763964.701486:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763964.701493:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3939 00000100:00080000:2.0:1789763964.701498:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763964.701505:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763964.701512:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763964.701529:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699513104128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763964.701553:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513103744:192.168.203.156@tcp:250:kworker.0 00000800:00000100:1.0:1789763964.701658:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763964.701733:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763964/real 1789763964] req@ffff8dc00368fb80 x1876699513103744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763964.701769:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699513103744/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763964.701786:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763964.701791:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763964.701797:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513103744:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763964.701807:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513103872:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763964.701834:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763964.701849:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763964/real 1789763964] req@ffff8dc00368f480 x1876699513103872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763964.701864:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699513103872/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763964.701875:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763964.701879:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763964.701883:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513103872:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763964.701891:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513104000:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763964.701972:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763964.701988:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763964/real 1789763964] req@ffff8dc00368f100 x1876699513104000/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763964.702003:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699513104000/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763964.702013:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763964.702017:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763964.702021:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513104000:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763964.702055:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513104128:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763964.702078:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763964.702091:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763964/real 1789763964] req@ffff8dc00368d880 x1876699513104128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763964.702105:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699513104128/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763964.702115:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763964.702119:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763964.702122:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513104128:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763965.725187:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763966.749188:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763967.773104:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763968.797118:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763969.821070:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763969.821083:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763969.821090:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763969.821103:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3944 00010000:00010000:0.0:1789763969.821114:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763969.821120:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763969.821133:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763969.821197:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699513104256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763969.821246:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763969.821251:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763969.821254:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763969.821261:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3944 00000100:00080000:3.0:1789763969.821268:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763969.821275:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763969.821288:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763969.821302:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513104256:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763969.821314:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699513104384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763969.821331:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763969.821336:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763969.821339:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763969.821344:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3944 00000100:00080000:3.0:1789763969.821350:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763969.821354:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763969.821361:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763969.821384:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699513104512/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763969.821400:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789763969.821447:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763969.821456:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763969.821461:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763969.821473:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3944 00000100:00080000:3.0:1789763969.821482:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763969.821492:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763969.821503:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763969.821504:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763969/real 1789763969] req@ffff8dc0107af800 x1876699513104256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763969.821547:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699513104256/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763969.821547:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699513104640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1789763969.821616:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763969.821624:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763969.821633:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513104256:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763969.821667:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513104384:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763969.821701:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763969.821718:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763969/real 1789763969] req@ffff8dc0107ae680 x1876699513104384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763969.821734:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699513104384/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763969.821744:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763969.821748:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763969.821752:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513104384:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763969.821759:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513104512:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763969.821779:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763969.821791:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763969/real 1789763969] req@ffff8dc0107ace00 x1876699513104512/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763969.821805:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699513104512/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763969.821815:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763969.821819:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763969.821823:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513104512:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763969.821829:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513104640:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763969.821847:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763969.821858:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763969/real 1789763969] req@ffff8dc0107af100 x1876699513104640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763969.821881:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699513104640/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763969.821890:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763969.821894:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763969.821897:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513104640:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763970.845060:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763971.869058:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763972.893067:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763973.917050:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763974.941029:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763974.941045:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763974.941052:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763974.941067:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3949 00000100:00080000:3.0:1789763974.941084:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763974.941095:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789763974.941103:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763974.941158:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aea00 x1876699513104768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763974.941193:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763974.941197:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763974.941199:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763974.941204:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3949 00000100:00080000:3.0:1789763974.941210:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763974.941215:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763974.941229:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763974.941247:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad880 x1876699513104896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763974.941266:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513104768:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763974.941266:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763974.941291:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763974.941295:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763974.941301:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3949 00000100:00080000:3.0:1789763974.941306:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763974.941311:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763974.941318:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763974.941340:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af100 x1876699513105024/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763974.941356:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763974.941361:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763974.941363:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763974.941370:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3949 00000100:00080000:3.0:1789763974.941403:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763974.941409:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763974.941416:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763974.941426:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763974.941442:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ace00 x1876699513105152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763974.941526:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763974/real 1789763974] req@ffff8dc0107aea00 x1876699513104768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763974.941568:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aea00 x1876699513104768/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789763999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763974.941586:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763974.941595:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763974.941603:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513104768:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763974.941647:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513104896:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763974.941696:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763974.941716:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763974/real 1789763974] req@ffff8dc0107ad880 x1876699513104896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763974.941734:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad880 x1876699513104896/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763974.941755:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763974.941760:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763974.941764:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513104896:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763974.941774:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513105024:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763974.941798:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763974.941811:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763974/real 1789763974] req@ffff8dc0107af100 x1876699513105024/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763974.941826:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af100 x1876699513105024/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789763999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763974.941836:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763974.941840:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763974.941844:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513105024:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763974.941851:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513105152:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763974.941872:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763974.941884:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763974/real 1789763974] req@ffff8dc0107ace00 x1876699513105152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763974.941898:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ace00 x1876699513105152/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789763999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763974.941970:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763974.941975:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763974.941980:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513105152:192.168.203.156@tcp:38:kworker.0 00010000:00010000:0.0:1789763975.965029:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763976.989033:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763978.013040:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789763979.037017:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763980.061017:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763980.061025:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763980.061029:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00010000:00010000:1.0:1789763980.061031:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763980.061038:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3954 00000100:00080000:3.0:1789763980.061047:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763980.061053:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763980.061093:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae680 x1876699513105280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763980.061120:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763980.061123:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763980.061125:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763980.061128:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3954 00000100:00080000:3.0:1789763980.061131:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763980.061135:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763980.061142:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789763980.061154:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513105280:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763980.061155:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af800 x1876699513105408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763980.061166:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763980.061168:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763980.061169:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763980.061172:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3954 00000100:00080000:3.0:1789763980.061174:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763980.061176:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763980.061179:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763980.061189:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac700 x1876699513105536/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763980.061196:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763980.061198:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763980.061199:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763980.061202:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3954 00000100:00080000:3.0:1789763980.061206:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763980.061208:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763980.061212:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763980.061214:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763980.061222:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107adf80 x1876699513105664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763980.061259:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763980/real 1789763980] req@ffff8dc0107ae680 x1876699513105280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763980.061273:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae680 x1876699513105280/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763980.061281:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763980.061284:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763980.061287:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513105280:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763980.061304:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513105408:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763980.061319:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763980.061328:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763980/real 1789763980] req@ffff8dc0107af800 x1876699513105408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763980.061350:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af800 x1876699513105408/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763980.061357:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763980.061359:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763980.061361:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513105408:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763980.061365:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513105536:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763980.061382:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763980.061389:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763980/real 1789763980] req@ffff8dc0107ac700 x1876699513105536/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763980.061396:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac700 x1876699513105536/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763980.061402:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763980.061404:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763980.061406:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513105536:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763980.061409:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513105664:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763980.061418:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763980.061424:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763980/real 1789763980] req@ffff8dc0107adf80 x1876699513105664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763980.061431:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107adf80 x1876699513105664/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763980.061436:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763980.061438:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763980.061440:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513105664:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763981.085168:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763982.109128:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763983.133149:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763984.157061:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789763985.180999:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763985.181007:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789763985.181011:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763985.181025:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3959 00000100:00080000:3.0:1789763985.181033:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763985.181040:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:2.0:1789763985.181053:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1789763985.181075:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107aed80 x1876699513105792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763985.181095:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763985.181099:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763985.181100:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763985.181102:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3959 00000100:00080000:3.0:1789763985.181105:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763985.181108:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763985.181116:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763985.181125:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac380 x1876699513105920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763985.181129:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513105792:192.168.203.156@tcp:250:kworker.0 00000100:00080000:3.0:1789763985.181136:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763985.181140:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789763985.181141:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763985.181144:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3959 00000100:00080000:3.0:1789763985.181146:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763985.181148:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763985.181153:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789763985.181161:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad180 x1876699513106048/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789763985.181167:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789763985.181169:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789763985.181171:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789763985.181174:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3959 00000100:00080000:3.0:1789763985.181177:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789763985.181180:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789763985.181183:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1789763985.181192:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763985.181193:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107af480 x1876699513106176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763985.181233:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763985/real 1789763985] req@ffff8dc0107aed80 x1876699513105792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763985.181247:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107aed80 x1876699513105792/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763985.181255:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763985.181258:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763985.181261:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107aed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513105792:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763985.181279:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513105920:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763985.181292:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763985.181315:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763985/real 1789763985] req@ffff8dc0107ac380 x1876699513105920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763985.181325:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac380 x1876699513105920/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763985.181331:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763985.181333:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763985.181335:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513105920:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763985.181339:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513106048:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763985.181353:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763985.181360:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763985/real 1789763985] req@ffff8dc0107ad180 x1876699513106048/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763985.181367:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad180 x1876699513106048/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763985.181376:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763985.181378:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763985.181380:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513106048:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763985.181383:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513106176:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763985.181393:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763985.181399:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763985/real 1789763985] req@ffff8dc0107af480 x1876699513106176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763985.181406:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107af480 x1876699513106176/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763985.181411:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763985.181413:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763985.181415:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513106176:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763986.205057:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763987.229047:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763988.253140:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763989.277047:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763990.300989:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763990.300999:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763990.301006:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763990.301018:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3964 00000100:00080000:2.0:1789763990.301030:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763990.301038:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:1.0:1789763990.301063:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763990.301083:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368d880 x1876699513106304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763990.301109:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763990.301111:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763990.301113:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763990.301116:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3964 00000100:00080000:2.0:1789763990.301119:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763990.301122:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763990.301133:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763990.301145:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f100 x1876699513106432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789763990.301147:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513106304:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789763990.301170:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763990.301173:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763990.301174:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763990.301178:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3964 00000100:00080000:2.0:1789763990.301180:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763990.301182:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763990.301186:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763990.301194:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f480 x1876699513106560/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763990.301202:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763990.301205:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763990.301206:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763990.301210:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3964 00000100:00080000:2.0:1789763990.301214:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763990.301216:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763990.301219:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763990.301227:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368fb80 x1876699513106688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1789763990.301241:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763990.301296:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763990/real 1789763990] req@ffff8dc00368d880 x1876699513106304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763990.301312:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368d880 x1876699513106304/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763990.301320:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763990.301324:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763990.301327:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513106304:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789763990.301348:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513106432:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763990.301366:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763990.301375:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763990/real 1789763990] req@ffff8dc00368f100 x1876699513106432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763990.301384:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f100 x1876699513106432/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763990.301389:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763990.301391:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763990.301394:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513106432:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789763990.301398:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513106560:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789763990.301409:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763990.301415:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763990/real 1789763990] req@ffff8dc00368f480 x1876699513106560/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763990.301423:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f480 x1876699513106560/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763990.301428:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763990.301430:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763990.301431:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513106560:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789763990.301435:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513106688:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789763990.301446:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789763990.301452:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763990/real 1789763990] req@ffff8dc00368fb80 x1876699513106688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789763990.301460:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368fb80 x1876699513106688/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789763990.301465:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789763990.301466:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789763990.301468:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513106688:192.168.203.156@tcp:38:kworker.0 00010000:00010000:2.0:1789763991.325033:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763992.349019:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763993.373056:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1789763994.397055:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789763995.420970:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763995.420980:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789763995.420985:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763995.420996:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3970 00000100:00080000:2.0:1789763995.421006:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763995.421013:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789763995.421033:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789763995.421051:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ca80 x1876699513106816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763995.421079:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763995.421084:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763995.421087:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763995.421092:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3970 00000100:00080000:2.0:1789763995.421097:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763995.421102:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789763995.421110:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513106816:192.168.203.156@tcp:250:kworker.0 00000008:00100000:2.0:1789763995.421112:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789763995.421133:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ce00 x1876699513106944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763995.421146:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763995.421150:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789763995.421152:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763995.421155:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3970 00000100:00080000:2.0:1789763995.421167:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763995.421173:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000800:00000100:3.0:1789763995.421175:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000008:00100000:2.0:1789763995.421180:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1789763995.421196:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368c000 x1876699513107072/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789763995.421203:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789763995.421206:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789763995.421207:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789763995.421210:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3970 00000100:00080000:2.0:1789763995.421267:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789763995.421271:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789763995.421277:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789763995.421286:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763995/real 1789763995] req@ffff8dc00368ca80 x1876699513106816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789763995.421293:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ea00 x1876699513107200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789763995.421307:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ca80 x1876699513106816/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763995.421320:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763995.421324:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763995.421327:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513106816:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789763995.421346:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513106944:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763995.421364:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763995.421373:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763995/real 1789763995] req@ffff8dc00368ce00 x1876699513106944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763995.421381:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ce00 x1876699513106944/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763995.421386:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763995.421389:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763995.421391:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513106944:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789763995.421394:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513107072:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789763995.421406:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763995.421412:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763995/real 1789763995] req@ffff8dc00368c000 x1876699513107072/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763995.421419:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368c000 x1876699513107072/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763995.421433:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763995.421437:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763995.421441:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513107072:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789763995.421450:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513107200:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789763995.421473:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789763995.421488:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789763995/real 1789763995] req@ffff8dc00368ea00 x1876699513107200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789763995.421497:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ea00 x1876699513107200/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789763995.421502:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789763995.421504:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789763995.421507:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513107200:192.168.203.156@tcp:38:kworker.0 00010000:00010000:1.0:1789763996.445039:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763997.469079:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789763998.493075:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1789763999.517075:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1789764000.541021:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764000.541030:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1789764000.541034:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789764000.541043:0:11858:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3975 00000100:00080000:2.0:1789764000.541054:0:11858:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789764000.541060:0:11858:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00010000:00010000:3.0:1789764000.541069:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1789764000.541096:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368ed80 x1876699513107328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789764000.541117:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764000.541120:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789764000.541122:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789764000.541126:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3975 00000100:00080000:2.0:1789764000.541130:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789764000.541135:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789764000.541144:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1789764000.541161:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368e300 x1876699513107456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1789764000.541170:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513107328:192.168.203.156@tcp:250:kworker.0 00000100:00080000:2.0:1789764000.541183:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764000.541244:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1789764000.541246:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789764000.541251:0:11858:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3975 00000100:00080000:2.0:1789764000.541258:0:11858:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1789764000.541264:0:11858:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789764000.541270:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1789764000.541275:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:2.0:1789764000.541293:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368f800 x1876699513107584/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1789764000.541307:0:11858:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1789764000.541310:0:11858:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1789764000.541313:0:11858:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1789764000.541319:0:11858:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3975 00000100:00080000:2.0:1789764000.541325:0:11858:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1789764000.541330:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764000/real 1789764000] req@ffff8dc00368ed80 x1876699513107328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1789764000.541332:0:11858:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:2.0:1789764000.541348:0:11858:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789764000.541358:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368ed80 x1876699513107328/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1789764000.541366:0:11858:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc00368df80 x1876699513107712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789764000.541381:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789764000.541389:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789764000.541395:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513107328:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789764000.541428:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513107456:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789764000.541454:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789764000.541466:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764000/real 1789764000] req@ffff8dc00368e300 x1876699513107456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789764000.541475:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368e300 x1876699513107456/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789764000.541490:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789764000.541493:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789764000.541495:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513107456:192.168.203.156@tcp:38:kworker.0 00000100:00100000:3.0:1789764000.541500:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513107584:192.168.203.156@tcp:8:kworker.0 00000800:00000100:3.0:1789764000.541516:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789764000.541523:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764000/real 1789764000] req@ffff8dc00368f800 x1876699513107584/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789764000.541531:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368f800 x1876699513107584/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789764000.541537:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789764000.541539:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789764000.541541:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513107584:192.168.203.156@tcp:8:kworker.0 00000100:00100000:3.0:1789764000.541545:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513107712:192.168.203.156@tcp:38:kworker.0 00000800:00000100:3.0:1789764000.541562:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:3.0:1789764000.541573:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764000/real 1789764000] req@ffff8dc00368df80 x1876699513107712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1789764000.541603:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc00368df80 x1876699513107712/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1789764000.541610:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1789764000.541612:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:3.0:1789764000.541614:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc00368df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513107712:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789764001.565133:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789764002.589058:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789764003.613102:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789764004.637059:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789764005.661057:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) e6cf7267-9b19-462a-a263-14115df15dab->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789764005.661068:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1789764005.661072:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789764005.661085:0:12051:0:(import.c:510:import_select_connection()) MGC192.168.203.156@tcp: connect to NID 192.168.203.156@tcp last attempt 3980 00010000:00010000:0.0:1789764005.661090:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1789764005.661096:0:12051:0:(import.c:589:import_select_connection()) MGC192.168.203.156@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789764005.661104:0:12051:0:(import.c:624:import_select_connection()) MGC192.168.203.156@tcp: import ffff8dc010461800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000100:00100000:3.0:1789764005.661164:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ad500 x1876699513107840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789764005.661194:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789764005.661199:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789764005.661201:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789764005.661206:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3980 00000100:00080000:3.0:1789764005.661211:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789764005.661217:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc013c56000: import ffff8dc0127ce800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789764005.661230:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1789764005.661240:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513107840:192.168.203.156@tcp:250:kworker.0 00000100:00100000:3.0:1789764005.661251:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ac000 x1876699513107968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789764005.661263:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) 50a9c642-d267-4bb6-8271-a3ab7ad8230f->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789764005.661266:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1789764005.661268:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1789764005.661271:0:12051:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: connect to NID 192.168.203.156@tcp last attempt 3980 00000100:00080000:3.0:1789764005.661274:0:12051:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789764005.661279:0:12051:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8dc013c56000: import ffff8dc0127cd000 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789764005.661284:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683244 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1789764005.661293:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107afb80 x1876699513108096/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1789764005.661301:0:12051:0:(pinger.c:229:ptlrpc_pinger_process_import()) d2c40150-e97b-40da-952b-cb9f18bf7e96->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1789764005.661304:0:12051:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1789764005.661305:0:12051:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1789764005.661311:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00080000:3.0:1789764005.661311:0:12051:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: connect to NID 192.168.203.156@tcp last attempt 3980 00000100:00080000:3.0:1789764005.661320:0:12051:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1789764005.661325:0:12051:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8dc003505800: import ffff8dc003581800 using connection 192.168.203.156@tcp/192.168.203.156@tcp 00000008:00100000:3.0:1789764005.661331:0:12051:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683244 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1789764005.661344:0:12051:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8dc0107ae300 x1876699513108224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1789764005.661371:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764005/real 1789764005] req@ffff8dc0107ad500 x1876699513107840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764005.661387:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ad500 x1876699513107840/t0(0) o250->MGC192.168.203.156@tcp@192.168.203.156@tcp:26/25 lens 520/544 e 0 to 1 dl 1789764030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764005.661396:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc010461800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764005.661399:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789764005.661402:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:e6cf7267-9b19-462a-a263-14115df15dab:2357:1876699513107840:192.168.203.156@tcp:250:kworker.0 00000100:00100000:1.0:1789764005.661422:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513107968:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789764005.661437:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789764005.661445:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764005/real 1789764005] req@ffff8dc0107ac000 x1876699513107968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764005.661458:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ac000 x1876699513107968/t0(0) o38->lustre-MDT0000-mdc-ffff8dc013c56000@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764005.661464:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127ce800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764005.661466:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789764005.661468:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513107968:192.168.203.156@tcp:38:kworker.0 00000100:00100000:1.0:1789764005.661472:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513108096:192.168.203.156@tcp:8:kworker.0 00000800:00000100:1.0:1789764005.661483:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789764005.661490:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764005/real 1789764005] req@ffff8dc0107afb80 x1876699513108096/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764005.661497:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107afb80 x1876699513108096/t0(0) o8->lustre-OST0000-osc-ffff8dc013c56000@192.168.203.156@tcp:28/4 lens 520/544 e 0 to 1 dl 1789764030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764005.661502:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc0127cd000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764005.661504:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789764005.661506:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107afb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:50a9c642-d267-4bb6-8271-a3ab7ad8230f:2357:1876699513108096:192.168.203.156@tcp:8:kworker.0 00000100:00100000:1.0:1789764005.661509:0:2357:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513108224:192.168.203.156@tcp:38:kworker.0 00000800:00000100:1.0:1789764005.661520:0:2357:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.156@tcp 00000100:00100000:1.0:1789764005.661526:0:2357:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1789764005/real 1789764005] req@ffff8dc0107ae300 x1876699513108224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1789764005.661533:0:2357:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8dc0107ae300 x1876699513108224/t0(0) o38->lustre-MDT0000-mdc-ffff8dc003505800@192.168.203.156@tcp:12/10 lens 520/544 e 0 to 1 dl 1789764030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1789764005.661542:0:2357:0:(import.c:65:import_set_state_nolock()) ffff8dc003581800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1789764005.661544:0:2357:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.156@tcp failed (-110) 00000100:00100000:1.0:1789764005.661546:0:2357:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8dc0107ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:d2c40150-e97b-40da-952b-cb9f18bf7e96:2357:1876699513108224:192.168.203.156@tcp:38:kworker.0 00010000:00010000:3.0:1789764006.685035:0:5597:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1789764007.709047:0:9608:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1789764008.733009:0:5598:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 98023 lines, 98023 kept, 0 dropped, 0 bad.